Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
79ee3e3392 | ||
|
|
36ef8dcdec | ||
|
|
ade357a14e | ||
|
|
a86ab64f6c | ||
|
|
1d4abca54a | ||
|
|
9647f1d158 | ||
|
|
67053335d1 | ||
|
|
3add386aa9 | ||
|
|
dcf4f84cb7 | ||
|
|
80f1ab3167 | ||
|
|
876d92367e | ||
|
|
77015997f8 | ||
|
|
44d1904286 | ||
|
|
6018486fcc | ||
|
|
01bf4e83a9 | ||
|
|
1a788f332d | ||
|
|
3363ffa89a | ||
|
|
0003115092 | ||
|
|
20fa831eaf | ||
|
|
a91d0b2421 | ||
|
|
af5c005a4e | ||
|
|
5277d859d5 | ||
|
|
8257ff1cbe | ||
|
|
87dc50743c | ||
|
|
4b9b0af3cb | ||
|
|
2c9152d912 | ||
|
|
52af38df6f | ||
|
|
34979d5d25 | ||
|
|
0bb57a4fee | ||
|
|
2aebf1bde1 | ||
|
|
d306b8c2d7 | ||
|
|
be7e97dc1a |
No files matched your search
@@ -38,7 +38,7 @@ to probe a refusal at all.**
|
||||
## If a probe does create something, remove it through the product
|
||||
|
||||
Not by hand, and not by `docker rm`: stop it, then `POST /api/stacks/<name>/remove` with
|
||||
`remove_hdd_data` and `remove_backups`, and then **verify** — no container, no `/opt/felhom/stacks/<name>`,
|
||||
`remove_hdd_data` and `remove_backups`, and then **verify** — no container, no `/opt/docker/stacks/<name>/app.yaml` (the folder itself stays — it holds the catalog's template files; path corrected 2026-10-24),
|
||||
no volume. Say in the report that it happened. A tidy-up nobody is told about is how the next session
|
||||
learns nothing.
|
||||
|
||||
|
||||
@@ -16,6 +16,117 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-25 (attended release run) — D49 = A, D44/D45 = all picks, D46 = yes; released: hub 0.170.0, agent 0.165.0,
|
||||
> controller 0.343.0, catalog `ebbf1f4`** (`09` §3 decisions 258–261; `REPORT-release-2026-10-26.md`). **D49 = A:** a
|
||||
> restore whose app does not start keeps both copies and never pours back by itself (`07` §8.2e). **Fix A** (controller
|
||||
> `restore_verdict.go`): „started" = running on every sample of 45 s, no `RestartCount` growth, every healthcheck
|
||||
> `healthy`; the undo copy is released only after 3 more minutes of the same (10 s samples, 30 min watch, a newer restore
|
||||
> ends the watch — `restoreBegunAfter`). **Fix B:** the `.fab` import pulls first (`Exporter.SetImagePreparer` →
|
||||
> `PrepareRestoreImages`). **D47 follow-up:** a set-aside mails the operator (`backup_run_failures`, leg
|
||||
> `unit-set-aside`) — its hub subject still says „failed to back up" (R-1287). Agent 0.165.0 vouched 18:38 CET; on a
|
||||
> BYO host our SSH drop-in is removed only on the bundle run AFTER the one that installs the new wrapper (R-1285).
|
||||
> Installer 1.36.1 on main, not published (R-1284: `http_code` printed the code twice). R-1250's live proof is a dated
|
||||
> check on 2026-10-27 (the agent tests only on its due Tuesday 10:00–15:59).
|
||||
|
||||
> **2026-10-25 08:38 — operator rulings by starting the R-1269 brief: D47 and D48** (`09` §3 decisions 256, 257;
|
||||
> `audits/restore-truth-2026-10-25/README.md`). **D47:** an existing recovery unit is never changed in part — its
|
||||
> definition and data are written only together, by a backup of the same install; a unit an earlier install left is set
|
||||
> aside whole and stays restorable (`07` §8.2d). **D48:** every restore path resolves and pulls every image its
|
||||
> definition names before the first change to the app; a failed pull is a refusal with the app untouched (`07` §6.6).
|
||||
> **Confirmed, no change:** `07` §6.6's version rule; R-698 and R-933. Built the same day, controller commits only (rides
|
||||
> v0.343.0): `02daebe` (D47 / R-1269), `b680db8` (R-1270: a restore whose app does not start is never a success; the
|
||||
> safest reading — no automatic pour-back, the undo copy kept, a late start turns it into a success), `2d71270` (D48),
|
||||
> `bd9f4f7` (R-1271: a copy dated more than 10 minutes in the future is never „the newest").
|
||||
|
||||
> **2026-10-24 (afternoon, attended) — two DooPlex backup alarms — DONE.** One cause: the 2026-10-23 window reboot
|
||||
> restarted `postgresql-2`; CNPG 1.29 sets the role `postgres` from the superuser secret, then from the
|
||||
> `bootstrap.initdb` owner secret (`owner: postgres`), so `postgres-superuser` is in force at every start; the dump
|
||||
> CronJob read `postgresql-superuser` → 28P01 from 16:00Z → `PostgresqlDumpStale`; the 00:20 Gitea push refused the
|
||||
> stale dump → `DooplexGiteaOffsiteStale`. **Operator rulings 2026-10-24:** (1) fix the CronJob only (homelab
|
||||
> `531023d`; the Cluster keeps two writers — homelab list); (2) apply the routing change by restarting Alertmanager
|
||||
> (its `envsubst` init container makes `/-/reload` useless; homelab `22d7697`: critical 4 h, warning 12 h, one mail per
|
||||
> backup group and per drive); (3) the R-1250 live proof may use DooPlex's k3s qBittorrent as the downloading peer
|
||||
> (one private LAN torrent in its container `/tmp`, deleted with its files). R-1250 committed (agent `29bcf44`,
|
||||
> `1e087bb`; controller `046b808`), not released. R-934 closed (the disk was pulled 2026-10-23). Catalog `7de554e`:
|
||||
> the box walk sends the drive itself as HDD_PATH (R-839). Report: `REPORT-dooplex-alarms-2026-10-24.md`.
|
||||
|
||||
> **2026-10-24 (attended release run) — DONE.** Hub 0.168.0 (05:39Z) and 0.169.0 (07:51Z) deployed; agent 0.164.0
|
||||
> released, delivered to the three test boxes and vouched by the operator; controller 0.342.0 the global floor (08:29Z,
|
||||
> MinAgent 0.158.1). ep0 runs `felhom-iperf3` (tunnel only). First line tests: Tester 1 94.3, demo-hp 94.4, demo-felhom
|
||||
> 92.6 Mbit/s; demo-felhom's qBittorrent limited to 46 Mbit/s. D40 in effect (dated check 2026-10-25, R-1243). Opened
|
||||
> R-1249 (the start-up VACUUM), R-1250 (a seeding qBittorrent is never tested). Report `REPORT-attended-release-2026-10-24.md`.
|
||||
|
||||
> **2026-10-24 07:22 / 07:26 — operator rulings by starting the attended release brief (D40, D41, D42; the line test).**
|
||||
> **D40 (R-1243) = stop the Longhorn backup of the hub's volume**; the encrypted 6-hourly copy to ep0 stays the hub's
|
||||
> off-site copy; the operator deleted the two old plain Longhorn backups himself (UI); the Longhorn UI cannot take the
|
||||
> volume out of the backup job, so the change is in a manifest. **D41 (R-1245) = 50 %** of the measured line for
|
||||
> qBittorrent's default upload limit. **D42 (R-1246) = the family is told only about the torrent limit**;
|
||||
> `app_unexplained_upload` / `app_unexplained_cpu` are operator-only. **Line measure (R-1247) = a weekly iperf3 upload
|
||||
> test from the box's host to an iperf3 server on ep0, inside the WireGuard tunnel only** (operator 07:26; his design of
|
||||
> 2026-10-23: „during a quiet window the box measures the upload bandwidth"); the passive floor stays as a lower bound.
|
||||
> This is the one change on ep0 the run may make (Tier 2 — the operator's yes is starting the run). CC still stops
|
||||
> before each hub deploy, before the global controller floor, and for the agent vouch.
|
||||
> **Two more answers in the same run (AskUserQuestion, 2026-10-24 ~08:2x CEST):** (1) **the agent speaks iperf3 itself**
|
||||
> (no host has the iperf3 program; nothing is installed on hosts) — agent `internal/linetest`; (2) **the live hub file's
|
||||
> old token bytes are accepted, not cleaned** (the start-up VACUUM did not finish on the Longhorn disk, R-1249; a clean-up
|
||||
> needs an 8–20 min hub stop, a trim and the volume's snapshots deleted; only root on DooPlex can read the volume, and it
|
||||
> holds the key; every copy that leaves is clean) — written into `05` §16.2 with the rule „a hand copy of hub.db is made
|
||||
> with VACUUM INTO, never cp/tar".
|
||||
|
||||
> **2026-10-23 19:22 — operator rulings by starting the weekend-night brief (R-1241's Felhom half; the upload watch).**
|
||||
> **Part 1 = R-1241's Felhom half:** seal R-975 (done in hub 0.168.0, not deployed — `05` §16.2,
|
||||
> `audits/hub-token-seal-2026-10-23/`); the hub's plain Longhorn copies are decide row R-1243 (CC's pick: take the hub
|
||||
> volume out of the Longhorn backup; ep0's encrypted copy is the backup). **Part 2 = the upload watch, shaped by the
|
||||
> operator:** the box measures its own upload line (first from what it already sees — the off-site backup's upload
|
||||
> rate — a short test at a quiet hour only if needed); on a sustained high upload it attributes the bytes to an app's
|
||||
> container and ASKS THE APP (qBittorrent's seeding, Plex's remote sessions); a mismatch is the alarm
|
||||
> (`app_unexplained_upload`, the same shape for CPU, `app_unexplained_cpu`) — operator only. qBittorrent gets a default
|
||||
> upload limit from the measured line; a limit the household set is respected. **Open for the operator, CC's pick in
|
||||
> STATUS:** the default limit's size (CC: 50 % of the measured upload) and who is told (CC: the family only about the
|
||||
> qBittorrent limit they can change; the unexplained alarms to the operator only — the R-388 rule).
|
||||
|
||||
> **2026-10-24 (night, unattended) — the restore and backup holes; operator ruling D43 = A**
|
||||
> (`audits/restore-holes-2026-10-24/README.md`). **Ruling D43 = A (operator, 2026-10-24 evening, by starting the run):**
|
||||
> the root-key-only sshd drop-in applies only in appliance mode; BYO hosts keep their SSH settings (`11` §1, `03`).
|
||||
> Fixed on main, nothing released: controller `bb2809d`/`062f9b4` (R-1251: a pending put-back blocks every start — a
|
||||
> restore hold `putback_pending`; the put-back runs first when the drive binds; a failed put-back holds; the copy synced
|
||||
> before the marker names it), `518a135` (R-1252: reconcile only under a valid classification), `c9b4b82` (R-1253: a
|
||||
> stop-first export refuses while another operation holds the marker); hub `dcf4f84c` (R-1254: escrow in one
|
||||
> transaction; image built locally, not pushed); agent `c8d3c17` (R-1257: D43). Proven: R-1251 on Tester 1 (power cut,
|
||||
> disk off, disk back), R-1252 and R-1253 on 9202. D44/D45 prepared: `audits/review-round-2-2026-10-24/D44-D45-picks.md`.
|
||||
|
||||
> **2026-10-24 (evening, unattended) — review round 2** (`audits/review-round-2-2026-10-24/README.md`). The 17 architecture
|
||||
> documents checked against `main`: facts corrected at `file:line` (EDITS.md), rulings and promises left for the operator
|
||||
> (R-1266, D45), undocumented decisions written as notes (R-1265, D44). New: **R-1251 (P1)** a cut restore of a drive-backed
|
||||
> app is put back over newer data (shown with the real `Recover`); **R-1252 (P2)** a rejected backup block makes the
|
||||
> second-drive reconcile delete the household's copies (reproduced on 9202). Controller `b01bb4b` (unreleased, next
|
||||
> v0.343.0): a media restart skips household-stopped apps (R-721); the FileBrowser SSO header file appears whole.
|
||||
|
||||
> **2026-10-23 ~12:00 — operator rulings by starting the D39 brief (`09` §3 decision 255; R-1239 closed).** **D39 = B:** a
|
||||
> `security_fix` ladder step — box walk on 9202 required, bench optional, taken by the night leg. **No operator word per
|
||||
> incident:** a public advisory id (CVE or GHSA) + its https:// notice is the authority; the session that writes the step
|
||||
> reports it to the operator after the push. The design (Claude.ai's recommendation) was taken as written; one fact in the
|
||||
> brief was checked and held (Plex's box fixture exists but is a `NoRoute` stub, so `seed: "none"` is right). Built:
|
||||
> catalog `3898593` (`ladder.py` `check_security` + `engine_majors_crossed` importing `check-engine-major.py`; the move gate
|
||||
> accepts it; `check-test-record.py` WARNs on a head with no bench; `upgrade-test.py --write-ladder --security`;
|
||||
> `box_walk.py --security` + `refresh_box_mirror()`), controller **0.341.0** (`legCandidate` takes `proven` or
|
||||
> `security_fix`; global floor 11:24 UTC with MinAgent 0.158.1, the operator's yes in chat). Proof:
|
||||
> `audits/security-lane-2026-10-23/`. Older controllers skip the step at night and take it on a hand press (proven).
|
||||
|
||||
> **2026-10-23 10:56 — operator ruling: Plex to the fixed version now, without the test record (Plex2Shell).**
|
||||
> Verbatim: *„bump the version to the newest in the app template immediately (as there are no customers yet), I want the
|
||||
> demo boxes on the newest version asap."* It skips `09` §3 decision 13's test record **once, for Plex only**. Done: catalog
|
||||
> `6cc317d` (Plex 1.41.4.9463-630c9f557 → 1.43.5.11029-6bf8af3f0, CVE-2026-96652/-96654/-96655/-96656), pushed
|
||||
> `--no-verify` past `check-test-record-move.py` (its refusal is in `app-catalog-felhom.eu/REPORT.md`); the only box running
|
||||
> Plex (demo-hp 9201) was checked for the attack's signs (none) and updated by the Update press; the guide re-checked
|
||||
> (`guide.checked_version` moved). Decision 13 itself is unchanged. Follow-ups: R-1238 (Plex's test record and fixture),
|
||||
> R-1239 (a fast lane for security fixes — the operator's decision, D39). Evidence: `audits/plex-demo-boxes-2026-10-23/`.
|
||||
|
||||
> **2026-10-23 morning — the operator's „go" for the day brief: the releases, D31 as built, D32–D36 (decide rows in `STATUS-HISTORY.md`).**
|
||||
> **Releases:** controller 0.340.0 (global floor 06:41 UTC, MinAgent 0.158.1); agent 0.163.0 (binary sha256 561b5316…, bundle 3afb88d9…, vouched by the operator 07:58 UTC; delivered to the three test boxes by signed jobs through a step bundle `0.163.0-step1` — the D4 drop-in is a new path, R-880); installer 1.36.0 (tag `installer-v1.36.0` → 83388c08, both webpage pins); hub 0.167.0 (sha256:177860a9…, 08:27 UTC); the website `0bb57a4f`.
|
||||
> **D31** = as built (the re-issue reads the newest deletion of a KEY-HOLDING host; an install that got the password but no recovery package is rotated without a press). **D32** = yes: the sheet is names and places only, one A4, HU + EN; the operator prints and fills it. **D33** = yes: `scripts/release-copy/` nightly 01:40, newest 3 image tags (controller, hub, samba) + 5 agent versions, skopeo `dir:` (digest-true — why not `docker save`: a re-push re-compresses), ep0 `/srv/felhom-release-copy` under the one limited user `felhom-relcopy` (rrsync), cap 3 GB enforced by the sender, a read-only registry token of its own. **D34** = yes: hub snapshots at 02/08/14/20 (catch-up > 6 h), DooPlex push :30 after, limit 7 h + a wait for a snapshot being written (R-1234: 2 h was wrong), alarm 13 h; ep0 retention unchanged (keep-daily 14, keep-weekly 8). **D35** = yes, after the release week (R-1228, dated 2026-10-30). **D36** = yes, both sentences, published.
|
||||
> **Added on the operator's word the same day (not soaked):** D34's hub half went into 0.167.0 (asked before the build). Record: `audits/day-2026-10-23/`, `REPORT-day-2026-10-23.md`.
|
||||
|
||||
> **2026-10-22 evening — operator rulings D29, D30, D31 (CC's picks accepted; the decide rows now in `STATUS-HISTORY.md`).**
|
||||
> **D29** = no: the controller repository stays public for now (R-925). **D30** = the agent makes a new WireGuard key and
|
||||
> registers it on the existing peer path (R-1204). **D31** = the hub skips a deleted install that never made a key when it
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# REPORT — attended release run 2026-10-24: the hub's tokens locked, the hub out of the open backup, the iperf3 line test, the watch released
|
||||
|
||||
Brief: „the hub's tokens locked, the hub out of the open backup, an iperf3 line test via ep0, then the upload watch
|
||||
released" (operator rulings 07:22 / 07:26, `CONTEXT.md`). Read first: `05` §16.2, `06` §4.5 / §5, `07`, `08` §9, the
|
||||
spike `audits/SPIKE-upload-watch-2026-10-23.md`, the felhom-build-deploy skill. Evidence: `audits/line-test-2026-10-24/`.
|
||||
|
||||
## Five lines
|
||||
|
||||
1. **Hub tokens:** sealed in hub 0.168.0 and still working (demo-hp is served the same tunnel and API tokens it runs; the
|
||||
template fetch works; the next ep0 copy holds 0 of 8 token values). The live file keeps old bytes in free space — your
|
||||
call: accepted, written down, with the rule „copy hub.db only with VACUUM INTO".
|
||||
2. **The hub is out of the open backup:** the hub volume reads `default: disabled`; the folder holds no copy of it (your
|
||||
two deletions worked; nothing else is left). Tonight's job is the last proof (dated check 2026-10-25).
|
||||
3. **The line test:** demo-hp measured **94.4 Mbit/s** (Tester 1 94.3, demo-felhom 92.6) — the night backup said „at
|
||||
least 10".
|
||||
4. **Live where:** hub 0.169.0 and controller 0.342.0 run on all three boxes (the global floor); agent 0.164.0 on all
|
||||
three (vouched); demo-felhom's qBittorrent is limited to 46 Mbit/s. Only these three boxes report to the hub.
|
||||
5. **You still do:** nothing for this run. Two rows wait for a design (below).
|
||||
|
||||
## What I asked you, and what you said
|
||||
|
||||
- Deploy hub 0.168.0 → yes. Deploy hub 0.169.0 → yes. Global floor 0.342.0 → yes. Agent vouch → done.
|
||||
- The live file's old bytes → accept and write down (my first estimate of „2 minutes" was wrong; the disk writes ~1.3
|
||||
MB/s, so a clean-up means an 8–20 minute hub stop).
|
||||
- No host has the iperf3 program → the agent speaks iperf3 itself (nothing installed on hosts).
|
||||
|
||||
## Versions, digests, CI (by commit)
|
||||
|
||||
| Repo | Commit | What | CI |
|
||||
|---|---|---|---|
|
||||
| felhom.eu | `00031150` | hub 0.168.0 manifest + rulings | 2275 success |
|
||||
| felhom.eu | `3363ffa8` | D40: the PVC label | 2276 success |
|
||||
| felhom-agent | `f7e1dcb` | the line test | 2277 success |
|
||||
| felhom-controller | `ed1011b` | the tested line in the watch | 2278 success |
|
||||
| felhom.eu | `1a788f33` | docs, register, ep0 unit, hub `line_test` model | 2279 success |
|
||||
| felhom-controller | `42cda07` | the start-line wording | 2283 success |
|
||||
| felhom-agent | `08ca678` | CHANGELOG released, REPORT | (docs) |
|
||||
| felhom.eu | `01bf4e83` | hub 0.169.0 manifest | (see the final push) |
|
||||
| felhom-controller | `bddc477` | CHANGELOG released, REPORT | (docs) |
|
||||
|
||||
- **Hub 0.168.0** `sha256:3bfdecaa…` (from `5277d859`), deployed 05:39Z. **Hub 0.169.0** `sha256:fb9b2489…` (from
|
||||
`1a788f33`), deployed 07:51Z; 30-minute watch: only info events, 0 errors.
|
||||
- **Agent 0.164.0**: binary `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`, bundle
|
||||
`1d7b54d39d0c1aff682ba0c0a7f91aea0f9407a10d2f5a8bbfdc858d32c13341`, tag `v0.164.0`; signed `agent_update` to the three
|
||||
boxes, self-check 76/76; vouched (hub page + `Artifact manifest set` line).
|
||||
- **Controller 0.342.0** `sha256:d002d3b7…` (from `42cda07`); rc.2 on 9202; per-box floors → healthy on all three, box
|
||||
checks and https-only green; global floor 08:29:19Z, MinAgent 0.158.1, read back on the hub log and page.
|
||||
- **ep0:** `felhom-iperf3.service` (iperf3 3.18-2+deb13u2, already installed since 2026-07-03), one nftables line;
|
||||
no answer on the internet (`:22` as the control), an answer in the tunnel. CVEs: four fixed in this build; two open
|
||||
„minor" server DoS issues bounded by tunnel-only, TCP-only and systemd caps (`scripts/ep0-iperf3/README.md`).
|
||||
|
||||
## Rows
|
||||
|
||||
**Before 194, after 191; opened 2 (R-1249, R-1250), closed 5 (R-975, R-1244, R-1245, R-1246, R-1247).** R-1243 →
|
||||
VERIFY (dated check 2026-10-25); R-1241 keeps only the homelab half.
|
||||
|
||||
## Observations
|
||||
|
||||
- The start-up VACUUM after a legacy seal cannot finish on the Longhorn disk; its comment said 2.8 s — FILED: R-1249.
|
||||
- A qBittorrent that seeds all day keeps the uplink busy, so that box is never tested and gets no limit — FILED: R-1250.
|
||||
- The live hub file's old token bytes — NOT-A-FINDING: accepted by the operator 2026-10-24 (`05` §16.2), every copy that
|
||||
leaves is clean.
|
||||
- Two pre-deploy hub snapshots in `/data/snapshots` held plain tokens — NOT-A-FINDING: they roll off (one already pruned
|
||||
at 06:06Z; the other at the 12:00Z snapshot); same volume, same reader as above.
|
||||
- The cross-repo wire gate refused the agent's new `line_test` field until the hub modelled it — fixed in the session
|
||||
(hub 0.169.0 `sysfacts.LineTest`, the System page cell).
|
||||
- My first ArgoCD sync used a wrong commit id — NOT-A-FINDING: it failed to fetch and changed nothing; redone.
|
||||
- 9202 has no agent local API, so its controller cannot read a line test — NOT-A-FINDING: by design (scratch guest);
|
||||
the qBittorrent proof moved to demo-felhom, where the product path runs.
|
||||
- demo-hp's first watch minute could not resolve `plex` — NOT-A-FINDING: the known start-up race (2026-10-23).
|
||||
- The family's line on demo-felhom's qBittorrent page was not seen live — NOT-A-FINDING: the household's password is not
|
||||
CC's; the render is pinned by a test, and the text with the live value is in the evidence.
|
||||
- Old rc controller images (other sessions) sit on 9202 — NOT-A-FINDING: a scratch guest; mine (rc.2) is removed.
|
||||
- The hub's database is 530 MB, mostly report history (`host_reports` 305 MB, `reports` 128 MB) — NOT-A-FINDING here:
|
||||
it is why a VACUUM is slow, recorded in R-1249.
|
||||
|
||||
## Teardown, three layers
|
||||
|
||||
- **Machine:** 9202 — the throwaway qBittorrent removed through the product (0 containers, 0 volumes), back on the
|
||||
0.342.0 release, my rc.2 image removed. demo-hp's host — the probe binary deleted, nothing running.
|
||||
- **Host / DooPlex:** the database copies I restored for the checks shredded; the 9202 password copy, the signed job
|
||||
envelopes and the release clone shredded / removed. ep0 — the one planned change stays (`felhom-iperf3`); the backup
|
||||
of the firewall file is `/etc/nftables.conf.bak-20261024-iperf3`.
|
||||
- **Hub:** the three per-box floors cleared; the global floor 0.342.0 stays (the release).
|
||||
|
||||
## Instruction files, poster
|
||||
|
||||
No instruction file edited. System poster: a text-only change (ep0 also runs the tunnel-only line-test server) — the
|
||||
facts file and the matching poster text edited in the same commit.
|
||||
@@ -0,0 +1,67 @@
|
||||
# REPORT — every catalog app on a fresh box, the way a family uses it (2026-10-24 → 25 night, unattended)
|
||||
|
||||
**Plain words first.**
|
||||
- 65 apps: **52 work, 10 work with a problem, 0 broken**; 2 are not offered for a new install (by design); Wanderer
|
||||
cannot run on a box without DNS (known, R-739).
|
||||
- **The three worst:** (1) after „remove — keep backups" and a reinstall, restoring that backup breaks the app — the
|
||||
household's item is gone from view (Tandoor, AdventureLog; R-1269, P2); (2) that restore still says it worked (R-1270,
|
||||
P2); (3) „use my kept data" loaded another box's newer-dated database for Paperless-ngx, Nextcloud and Immich, and
|
||||
Immich never answered again (R-1271, P2). In all three the data is not deleted.
|
||||
- **Fixes made in the drill catalog, not live:** n8n stops sending usage data out; OpenGist gets 256 MB (at 128 MB one
|
||||
sign-in hit the limit 443 times and swapped 98 MB). Both proven on the sweep box pointed at the drill.
|
||||
- **Morning:** say yes to the two catalog fixes (in Monday's release list); decide D46 (Homepage behind the family gate —
|
||||
CC's pick: yes); delete hub host Tester4-aa9466.
|
||||
|
||||
**Safest readings taken (the brief allowed no question):** ISO 1.30.0 (the current public one, not 1.36.0); 12 GiB VM
|
||||
(the brief's minimum; demo-hp had 14 free); the kept-data question exists only for drive-folder apps, so volume apps
|
||||
were checked by a unit restore after the reinstall (that made a unit restore on every seeded app, not every fifth);
|
||||
„Mentés most" is local-only by design; backup/storage findings filed in their own category, not „Apps & catalog". All in
|
||||
the audit's head.
|
||||
|
||||
**Table and evidence:** `documentation/audits/catalog-sweep-2026-10-24/README.md` (the 65-row table, the findings, the
|
||||
R-1269 proof, D22 put-back PASS on PrivateBin and Vikunja, the drill proofs, teardown).
|
||||
|
||||
**Architecture read:** `07-backup-architecture.md` §6.5 (kept data) and §8, `16-storage-physical-identity.md` §5 (the
|
||||
internal place is the default drive — F3 NOT-A-FINDING), `10-localisation.md`.
|
||||
|
||||
**Two mistakes of this session, both caught and kept in the evidence:**
|
||||
1. The first drill pass ran on the live templates: I had believed the drill repo public; the box's clone failed and the
|
||||
controller kept its templates. Re-run through the box's local mirror (R-992's method) with a positive check before
|
||||
any install (`DRILL-CONFIRMED`). Memory note written.
|
||||
2. I reverted the OpenGist fix on a wrong reading („it grows into its limit"); an automated security review flagged the
|
||||
lowered cap; a cgroup measurement proved the fix needed and it went back in.
|
||||
|
||||
**Commits:** drill catalog (never live) `a601950` (n8n), `fd23571` → `c420053` (wrong revert) → `405077f` + `492a17f`
|
||||
(OpenGist). felhom.eu: this commit (audit + tools + evidence, register, STATUS, STATUS-HISTORY, capability map B1 +
|
||||
detail). CI by hash: in the final report.
|
||||
|
||||
**Register:** at the sweep's start 190 (other sessions filed 18 since); before this commit 208, after 217; opened 9
|
||||
(R-1269–R-1277), closed 0. Notes added to R-739 (Wanderer, no DNS) and R-1267 (anonymous volumes: OnlyOffice and RomM
|
||||
too; 52 left after every app was removed with data).
|
||||
|
||||
**Observations:**
|
||||
- FILED: R-1269 (reinstall's capture rewrites the kept unit), R-1270 (restore says ok while the app never starts),
|
||||
R-1271 (kept data loads another box's future-dated database), R-1272 (Homepage has no gate — operator, D46), R-1273
|
||||
(registry rate limit shown as „exit code 1"), R-1274 (restore refusal advice when tiers exist but have not copied),
|
||||
R-1275 (drive names in Hungarian on English pages), R-1276 (empty parent folder after remove with data), R-1277 (box
|
||||
fixtures drifted: OpenGist, Calibre-Web, Mealie, Papra, Rallly, Vaultwarden 1.37).
|
||||
- FIXED (drill, morning list): n8n usage data (`a601950`); OpenGist memory (`405077f`, `492a17f`).
|
||||
- NOTE on existing rows: R-739 (Wanderer), R-1267 (anonymous volumes).
|
||||
- NOT-A-FINDING: F3 install default is the internal place (`16` §5, by design); F4 English guide labels (the app's own
|
||||
words; the Hungarian copy is frozen by the copy gate); F5 Kimai in English (a Hungarian browser lands on `/hu/`); F6
|
||||
AdventureLog 95 % (docker stats with cache; anon 75 %); database first-start lines in SparkyFitness and Dawarich (no
|
||||
restart, item back); Radicale IP redirect (curl without Host); NZBGet/Transmission basic-auth prompt in screenshots (the
|
||||
apps' own login); Vaultwarden stranger 200 (its own login page; sign-up closed).
|
||||
|
||||
**Teardown:** machine — VM 9360 and LXC 9361 (both this session's) destroyed with `--purge`, no disk left on
|
||||
`nvme-scratch`; Tester 1, 9201, 9202, the thin pool and the ISO untouched; the box's logs copied off first. Host —
|
||||
DooPlex: the session's secret files shredded. Hub — host Tester4-aa9466 left for the operator to delete. ep0 — listed,
|
||||
nothing removed: the host's PBS link was set up, no whole-box copy went to it; the app copies went to the Tester4 restic
|
||||
repository on the storage box.
|
||||
|
||||
**Secrets:** read by name through `sec`; the evidence was scanned for every known secret value (positive control seen) and
|
||||
for token shapes; one Actual Budget session token (a throwaway app) was blanked; screenshots spot-checked (passwords sit
|
||||
behind „Show", never pressed). No password, key or token is in this commit.
|
||||
|
||||
**System poster:** no fact on it changed. **Instruction files:** none edited. **No** release, deploy, floor, vouch,
|
||||
publish or live catalog push.
|
||||
@@ -0,0 +1,103 @@
|
||||
# REPORT — day 2026-10-23: the releases and the DooPlex hour (the operator's „go")
|
||||
|
||||
Design read: `07-backup-architecture.md` §6.3/§8, `09-update-architecture.md` §3 decision 250, `03-host-agent.md`,
|
||||
`05-hub-architecture.md` §16.3, `runbooks/total-loss-of-dooplex.md`, `audits/dooplex-survival-2026-10-22/PLAN.md`.
|
||||
Evidence: `documentation/audits/day-2026-10-23/01`–`14`.
|
||||
|
||||
## Five lines
|
||||
|
||||
1. Released: controller 0.340.0 (global floor), agent 0.163.0 (vouched by the operator), installer 1.36.0, hub 0.167.0,
|
||||
the website with D36's two sentences.
|
||||
2. Root SSH from the LAN is key-only on all three test boxes; new installs get it from installer 1.36.0.
|
||||
3. The hub database now leaves every 6 hours (restored back byte-identical); the released images and agent packages
|
||||
leave nightly to ep0 (pulled back, every sha256 OK, one image loaded on a scratch box).
|
||||
4. One false failure mail at 10:28 (my install of the 6-hourly copy) — filed R-1234, fixed and red-proved the same hour.
|
||||
5. Register 191 → 184; the operator: print the sheet, Hosts page clean-up, D37 + D38.
|
||||
|
||||
## Per box — versions before → after, and the SSH checks
|
||||
|
||||
| Box | controller | agent | root-file bundle | `sshd -T` root | sshd reloaded | key login | felhom-sshd | swap = RAM (9201) |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| Tester 1 | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 (via 0.163.0-step1) | without-password | yes, 09:27:38 | ok | active, `sshd -t` ok | 4096/4096 |
|
||||
| demo-hp | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:19:44 | ok | active, `sshd -t` ok | 25898/25898 |
|
||||
| demo-felhom | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:24:42 | ok | active, `sshd -t` ok | 12288/12288 |
|
||||
| hub | 0.165.0 → 0.167.0 | | | | | | | |
|
||||
|
||||
Not proven (said plainly): the real root password over SSH (CC does not hold it — a dummy one is refused, but OpenSSH logs
|
||||
only „Failed password"); a felhom-sshd connection (it admits only the operator's tunnel peer); the physical console (the
|
||||
drop-in does not touch it; `passwd -S root` = P).
|
||||
|
||||
## Digests, sha256, CI by commit
|
||||
|
||||
| What | Value |
|
||||
|---|---|
|
||||
| controller 0.340.0 | `sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621` — `caaa86e`, CI 2245 success |
|
||||
| agent 0.163.0 binary / bundle | `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c` / `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8` — `1109092` CI 2246, `b4721ee` CI 2249 success |
|
||||
| step bundle 0.163.0-step1 | `6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf` |
|
||||
| installer 1.36.0 (served = tag) | `de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec` — `83388c08` CI 2247 success; pins `2aebf1bd` |
|
||||
| hub 0.167.0 | `sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0` — built at `d306b8c2` (hub/ = `be7e97dc`, CI 2250 success; `d306b8c2` CI 2251 success); manifest `34979d5d` |
|
||||
| website | `0bb57a4f` |
|
||||
| D34 fix (R-1234) | `52af38df` |
|
||||
| prometheus rules | homelab-manifests `1bbef892` |
|
||||
|
||||
CI of the last commits (`2aebf1bd`, `0bb57a4f`, `34979d5d`, `52af38df` and this close-out): see the final message.
|
||||
|
||||
## Parts
|
||||
|
||||
- **1 — controller 0.340.0:** 9202 smoke, per-customer floors, three boxes green, box walk + https-only rc 0, global floor
|
||||
06:41 UTC (MinAgent 0.158.1), read back with the WAL (`01`–`06`).
|
||||
- **2 — agent + installer:** release, a step bundle because the D4 drop-in is a new path (R-880), three signed jobs per
|
||||
box (`07`); the operator's vouch read back from the DB and the live page (`10`); installer tag, both pins, a
|
||||
webpage-only sync, the served script = the tag (`11`).
|
||||
- **3 — hub 0.167.0:** built from a clean clone; on the operator's word it also carries D34's 6-hourly snapshot (asked
|
||||
before the build; not soaked); hub-only sync; 30-minute watch: 0 hub events, 0 errors, all boxes reporting, no hub mail
|
||||
(`14`).
|
||||
- **4 — website:** D3/D5 patch + D36 HU + EN; site gates; published; read back live: new text present, old gone (control:
|
||||
the previous commit), screenshots at 390 and 1366 px, no sideways scroll (`12-website-readback/`).
|
||||
- **5 — D34:** retention unchanged, server-side on ep0 (`prune-operator-hubdb` 03:45, keep-daily 14, keep-weekly 8 — the
|
||||
current day's up to 4 copies until the next prune). Hand run 501.6 MiB / 24.5 MiB uploaded in 4 s; the newest copy
|
||||
restored from ep0 with the read-only token = the pod's snapshot by sha256 `a9ff4c8a…`, integrity ok, 4 hosts, holds
|
||||
today's vouch. Alarm 13 h. Runbook updated (`13`).
|
||||
- **6 — D33:** ep0 before: `/` 14 GB free (the data disk's root IS the PBS datastore — not used). One user
|
||||
`felhom-relcopy` (locked, rrsync-confined: shell and `..` refused), one directory `/srv/felhom-release-copy`. Cap 3 GB,
|
||||
enforced by the sender (ep0 has no quota). **skopeo, not `docker save`:** `dir:` keeps the registry's bytes, so a push
|
||||
back keeps the digest; no Docker daemon. A read-only registry token of its own, minted in the Gitea pod straight into a
|
||||
0600 file. First run 676 MB, 91 s; pulled back: 141 files OK, 9 images re-verified (digests match independent records),
|
||||
controller 0.340.0 loaded on 9202 (ID = config digest). Alarm `FelhomReleaseCopyStale`. Restore steps in
|
||||
`total-loss-of-dooplex.md` (`08`).
|
||||
- **7 — D32:** the sheet is names and places only, HU + EN, one A4 (rendered: 1 page; a copy in the operator's home
|
||||
folder); every place checked to exist. demo-hp: three superseded ISOs removed by name, `/` 8.5 → 14 GB free (`09`).
|
||||
R-548: the box half done, the product fix proposed (R-921's pattern).
|
||||
- **8 — close-out:** CONTEXT rulings, STATUS (decide rows D32–D36 → history, D37/D38 new, the do list, D35 dated
|
||||
2026-10-30), REPORTs and CHANGELOGs; the system poster's facts file updated, its hub-DB label edited (text), and a new
|
||||
arrow asked for in STATUS („System poster needs a refresh").
|
||||
|
||||
## Register — 191 → 184; opened 4, closed 11
|
||||
|
||||
Opened: R-1234 (closed the same hour), R-1235, R-1236, R-1237. Closed: R-49, R-982, R-985, R-991, R-1204, R-1205,
|
||||
R-1225, R-1226, R-1227, R-1230, R-1234. Updated: R-1214 (released; dated check 2026-10-24), R-1187, R-1189, R-232,
|
||||
R-548, R-923, R-924, R-1228 (D35, dated 2026-10-30), R-1183's check text (four snapshot slots).
|
||||
|
||||
## Observations
|
||||
|
||||
- The D34 first push refused the hub's own catch-up and mailed admin@ — FILED: R-1234 (closed, fixed `52af38df`).
|
||||
- The release copy is proven to load, not to restore a registry; it follows the newest versions, not the floor/vouch; no
|
||||
golden — FILED: R-1235.
|
||||
- Gitea OOM-killed at its 1 GiB limit, the last at 06:41:10 UTC during the release — FILED: R-1236 (STATUS D37).
|
||||
- The night sweep's mail: one-time test-box codes in one session record — FILED: R-1237.
|
||||
- The hub DB snapshot is 501 MiB, not 24 MB — NOT-A-FINDING: 24 MB is the compressed upload, which is what ep0 stores.
|
||||
- felhom-sshd does not answer from the LAN — NOT-A-FINDING: its firewall admits only the operator peer (H1, by design).
|
||||
- A dummy root password logs „Failed password", not a policy refusal — NOT-A-FINDING: OpenSSH's behaviour under
|
||||
`prohibit-password`; the policy is proven by `sshd -T` and the reload.
|
||||
- The vouch left `artifact_wrapper_sha256` empty — NOT-A-FINDING: an optional field; the wrapper rides the vouched bundle.
|
||||
- Guests 9202, 9401 (demo-hp) and a transient 990000 (Tester 1) have swap ≠ RAM — NOT-A-FINDING: outside the felhom pool.
|
||||
- `known_hosts` holds an older key for 192.168.0.155 (demo-hp's guest address) — NOT-A-FINDING: guests are reached by
|
||||
`pct exec`; nothing was changed.
|
||||
- The D4 drop-in needed a step bundle — NOT-A-FINDING: the documented R-880 path for a new bundle path.
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
Machine: 9202 — the test image untagged, the tar removed. Host: demo-hp `/tmp` tar removed; three ISOs removed on purpose
|
||||
(Part 7); DooPlex: the read-back copies and the hub-DB scratch restore shredded/removed. Hub: nothing provisioned; the
|
||||
per-customer floors cleared after the global floor. Left on purpose: the ep0 user + directory, the DooPlex release-copy
|
||||
job, the 6-hourly hub-DB timer, both alarms.
|
||||
@@ -0,0 +1,111 @@
|
||||
# REPORT — two DooPlex backup alarms, every alarm the mail route delivers, and R-1250 (2026-10-24, attended)
|
||||
|
||||
Brief: „two DooPlex backup alarms — find why, fix, and check every other alarm the new mail route now delivers".
|
||||
Read first: `runbooks/total-loss-of-dooplex.md`, `runbooks/RUNBOOK-hub-db-offsite-backup.md`,
|
||||
`audits/dooplex-survival-2026-10-22/PLAN.md`, R-232, R-1227, R-1241, the two homelab rules and the Alertmanager
|
||||
route, homelab HM-040…HM-049 and the window record; design `08` §9. Evidence: `audits/dooplex-alarms-2026-10-24/`.
|
||||
|
||||
## Three lines
|
||||
|
||||
1. **Both alarms had one cause, and both are fixed.** Yesterday's reboot restarted the database, and it now takes the
|
||||
password of the other of its two secrets; the dump job could not log in, and the Gitea off-site push then refused
|
||||
the old dump. The job now reads the right secret. Cleared: the dump alarm 10:44Z, the Gitea alarm 10:51Z.
|
||||
2. **Nothing else fires now.** In 7 days only the pulled disk's two alarms fired long (silenced by you since 10-12,
|
||||
ended by the pull); the rest were minutes long or are known.
|
||||
3. **Fewer mails:** critical every 4 h (was 1 h), warning every 12 h (was 4 h), all backup alarms in one mail.
|
||||
|
||||
## What I asked you, and what you said
|
||||
|
||||
- The dump fix → **the CronJob only** (the database's own definition untouched).
|
||||
- The mail change needs an Alertmanager restart, not a reload → **restart**.
|
||||
- The R-1250 proof needed a downloader on another machine → **use DooPlex's k3s qBittorrent**.
|
||||
|
||||
## The alarm table (7 days, Prometheus `ALERTS`; nothing firing at 10:51Z)
|
||||
|
||||
| Alarm | Severity | When (UTC) | True? | Cause | Fix / row |
|
||||
|---|---|---|---|---|---|
|
||||
| `DooplexGiteaOffsiteStale` | critical | 10-24 00:57 → 10:51 | true | the push refused a 12 h old dump | the dump fix; R-232 note |
|
||||
| `PostgresqlDumpStale` | warning | 10-23 22:17 → 10-24 10:44 | true | dump job login failed (28P01) | homelab `531023d` |
|
||||
| `DiskOfflineUncorrectable`, `DiskUncorrectableSectors` (Z6A07P2G) | critical | whole window (≥ 145 h) → 10-23 12:02 | true | the dying HDD4 | silenced by you 10-12; pulled 10-23 — R-934 closed |
|
||||
| `DiskDisappeared` (Z6A07P2G) | critical | 10-23 12:17 → 18:02 | true | the pull | expected; cleared |
|
||||
| `DiskInventoryChanged` | warning | 10-23 12:17 → 20:17 | true | the pull | expected; cleared |
|
||||
| `LonghornVolumeDegraded` (two Plex volumes) | warning | 10-23 08:22 → 11:22 | true | the HM-040 rebuild | cleared |
|
||||
| `PodRestartingFrequently` (felhom-webpage) | warning | 10-19 09:02 → 10:07 | true | git-syncs stopped on errors | R-1163 (closed 10-19) |
|
||||
| `WanUploadDegraded`, `WanDownloadDegraded`, `WanUpstreamPacketLoss`, `WanBufferbloat` | warning/info | minutes, 10-17 → 10-20 | true | short WAN dips (the flood days) | none needed |
|
||||
| `AuthentikHighLatency` | warning | 10-21 08:12, 5 min | true | a short spike | none needed |
|
||||
|
||||
## The mail change (homelab `22d7697`, applied by a restart, read back from the running Alertmanager)
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| critical | every 1 h | every 4 h |
|
||||
| warning | every 4 h | every 12 h |
|
||||
| other (info) | every 4 h | every 12 h |
|
||||
| backup alarms | one mail stream per alarm | one mail for the group, every 4 h |
|
||||
| a disk | one stream per alarm | one mail per drive, daily (unchanged) |
|
||||
| resolved mail | `send_resolved: true` | unchanged; the mail counter rose once after each clear (10:47, and by 10:57) — that they read „RESOLVED" is not seen: the mailbox is not mine |
|
||||
|
||||
## Part 4 — R-1250, committed and proven
|
||||
|
||||
Agent `29bcf44` + `1e087bb`, controller `046b808` (commit only; release in the next train). A seeding box: the agent
|
||||
says „waiting for quiet"; the controller records the torrent app's limit, holds it at 1 KiB/s, asks for one test,
|
||||
puts the exact limit back. **Proven on demo-hp with a real seed** (10 Mbit/s from 9202 to DooPlex's qBittorrent):
|
||||
refused without the hold (11.1 Mbit/s); with it the uplink fell to 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, the limit
|
||||
back to 1 249 280 B/s; a kill in the middle → the start-up check put it back. The brief said „a nested VM": NOT done —
|
||||
a nested VM would need its own tunnel to ep0 to run the real test; demo-hp's host with probes did the same with the
|
||||
real tunnel. The pinned agent client is unit-tested only.
|
||||
|
||||
## Commits, CI, undo
|
||||
|
||||
| Repo | Commit | What | CI |
|
||||
|---|---|---|---|
|
||||
| homelab-manifests | `531023d` | the dump CronJob's secret | no CI in this repo |
|
||||
| homelab-manifests | `22d7697` | the mail routes, the dump alarm's label | no CI |
|
||||
| homelab-manifests | `9f399fe` | REPORT | no CI |
|
||||
| felhom-agent | `29bcf44`, `1e087bb` | R-1250 | 2286, 2288 success |
|
||||
| felhom-controller | `046b808` | R-1250 | 2287 success |
|
||||
| app-catalog-felhom.eu | `7de554e`, `5e196e1` | the box walk: HDD_PATH, the leftover check | 2289 success, (final push) |
|
||||
| felhom-agent / felhom-controller | `5e05627` / `321df1f` | REPORT, README | (final push) |
|
||||
| felhom.eu | this commit | register, STATUS, CONTEXT, `08` §9, evidence | (final push) |
|
||||
|
||||
Undo: `audits/dooplex-alarms-2026-10-24/README.md` § Undo. Backups of the three homelab files: `*.before` there.
|
||||
|
||||
## Rows
|
||||
|
||||
**Before 191, after 190; opened 0, closed 1 (R-934 — the disk was already pulled).** R-232: a dated note and a dated
|
||||
check (2026-10-25, tonight's sweep). R-1250: NARROWED (fixed on main, waits for the release train).
|
||||
|
||||
## Observations
|
||||
|
||||
- The database's two secrets for one role stay (only the CronJob moved) — homelab list item 1 (not a Felhom row).
|
||||
- A stale dump stops the Gitea + secrets off-site push for as long as it lasts — NOT-A-FINDING: the designed fail-closed
|
||||
order (`PLAN.md`: the dump first, for consistency); the alarm caught it inside its 26 h, and it now comes in one mail
|
||||
with its cause.
|
||||
- Alertmanager cannot reload a ConfigMap change (the brief assumed it could) — NOT-A-FINDING: the homelab manifest's
|
||||
design (`envsubst` init container); you chose the restart; the fact is in CC's memory and the homelab list.
|
||||
- The agent said „waiting for quiet" before it had watched a minute — fixed in the session (`1e087bb`, found live).
|
||||
- The box walk sent a folder inside the drive as HDD_PATH (refused since R-839) — fixed in the session (`7de554e`).
|
||||
- The box walk called the template folder a leftover after every remove — fixed in the session (`5e196e1`).
|
||||
- R-934 said the failed disk was still in DooPlex; STATUS still listed R-1214's done check — fixed (row closed, line removed).
|
||||
- CC's memory said Prometheus has no config-reloader; it has one since 2026-10-12 — fixed (memory note).
|
||||
- `govulncheck` warns GO-2026-6809 (crypto/x509, stdlib) in the agent and the controller — NOT-A-FINDING here: the Go
|
||||
toolchain row R-972 owns it.
|
||||
- My Gmail connector's mailbox gets no Alertmanager mail — NOT-A-FINDING: it is another mailbox than `SMTP_TO`; 60 sent,
|
||||
0 failed in 7 days. The Alertmanager config did not change yesterday (last 10-12).
|
||||
- `kubectl exec -i` never closed stdin for `pg_restore` — NOT-A-FINDING: my method; the host's own `pg_restore` read the dumps.
|
||||
- The 9202 ISO torrent never uploaded (behind NAT, 18 leechers / 1080 seeds) — NOT-A-FINDING: why the LAN peer was used.
|
||||
- I overwrote homelab-manifests' REPORT.md, which held another session's HM-050 report from today — NOT-A-FINDING: the
|
||||
repo's overwrite rule; HM-050's entry stays in its CHANGELOG.
|
||||
|
||||
## Teardown, three layers
|
||||
|
||||
- **Machine:** 9202 — qBittorrent removed through the product (no container, no volume; the template folder stays, as for
|
||||
every app); the test files, the forwarder and the probe removed. DooPlex's qBittorrent — the test torrent and its
|
||||
files deleted (by hash), nothing else touched.
|
||||
- **Host:** demo-hp — the agent probe stopped and deleted, no iperf3 left; the installed agent untouched. DooPlex — the
|
||||
9202 password copy and the helper's session files shredded; the clones removed.
|
||||
- **Hub:** nothing provisioned.
|
||||
|
||||
## Instruction files, poster
|
||||
|
||||
No instruction file edited. No poster fact changed (the mail intervals are not on it).
|
||||
@@ -0,0 +1,42 @@
|
||||
# REPORT — Plex to the fixed version in the catalog and on the demo/test boxes (2026-10-23 day, Plex2Shell)
|
||||
|
||||
**Plain words first.**
|
||||
- The catalog now gives Plex 1.43.5.11029-6bf8af3f0 (catalog `6cc317d`, pushed without the update test, on your ruling).
|
||||
- One box ran Plex: demo-hp. It showed no sign of the attack. CC pressed Update; it runs the new Plex now.
|
||||
- You have one decision (D39): may a security fix reach the boxes at night without the full update test?
|
||||
|
||||
## Per box
|
||||
|
||||
| Box | Plex before → after | Signs check | Box walk |
|
||||
|---|---|---|---|
|
||||
| demo-hp 9201 | 1.41.4.9463-630c9f557 → 1.43.5.11029-6bf8af3f0 | clean (NOT-A-FINDING; logs reach back only to 00:32 UTC today — CPU history and files cover 10-16 → 10-22) | green: all containers healthy, 0 errors, 0 warnings since the press |
|
||||
| demo-hp 9202, demo-felhom 9201, tester-1 9201 | no Plex | — | — |
|
||||
| Tester4 | hub host DOWN, VM gone | — | — |
|
||||
| Tester 2 | no host on the hub | — | — |
|
||||
|
||||
Detail and evidence: `documentation/audits/plex-demo-boxes-2026-10-23/README.md`.
|
||||
|
||||
## Records
|
||||
|
||||
- Register 184 → 186: opened R-1238 (P2, Plex has no test record or fixture), R-1239 (P2, a fast lane for security
|
||||
fixes — the operator's decision); closed none.
|
||||
- `CONTEXT.md`: the 10:56 ruling. `STATUS.md`: decide row D39. No „broken" line: Part 2 was finished before the
|
||||
records were written, so the line the brief asked for would have been added and removed in the same commit.
|
||||
The „Updated" paragraph was left as the day session wrote it (it is that session's summary; this run adds one
|
||||
decide row only).
|
||||
- Architecture read: `09-update-architecture.md` §3 decisions 13 and 52; `13-threat-model.md` (Plex row);
|
||||
`runbooks/target-selection.md`. Decision 13 is unchanged; no poster fact changed (none of the three poster actions).
|
||||
- `--no-verify` was used ONCE, for catalog `6cc317d`, on the ruling. The `check-test-record-move.py` refusal is verbatim
|
||||
in `app-catalog-felhom.eu/REPORT.md`.
|
||||
|
||||
## Fixed without a row
|
||||
|
||||
- The memory index (`.claude-memory/MEMORY.md`, outside git) had grown to 201 lines / 25 735 bytes at 11:09 (another
|
||||
session), over the 200 / 25 600 ceiling, so `instructions_gate` failed and so did two script suites that run it.
|
||||
Lines 71–72 (two memories about the deleted credentials file, both superseded by `sec`) became one line: 200 lines,
|
||||
25 590 bytes. `repo_gates.py --fast` then OK. The index sits right at the ceiling; the next added line trips it again.
|
||||
|
||||
## Teardown
|
||||
|
||||
Machine: Plex stays on the new version; helper scripts removed from guest and host. Host: nothing provisioned.
|
||||
Hub: reads only.
|
||||
@@ -0,0 +1,75 @@
|
||||
# REPORT — did the Plex attack on DooPlex reach Felhom's secrets? (2026-10-23)
|
||||
|
||||
## Plain words first
|
||||
|
||||
1. **Verdict: NOT REACHED.** Two open doors led from the hacked Plex to Felhom's secrets. Neither was used in the attack
|
||||
days: the backup folder got no new connection at all, and Longhorn made no backup or disk outside its normal plan.
|
||||
2. **The shared backup folder: closed to the home network and the tailnet** (your yes). It is **still open to every app
|
||||
on DooPlex** — Longhorn needs that, so the real fix is the homelab session's.
|
||||
3. **What you do next:** give the homelab session the hand-over list (`documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md`,
|
||||
last section). Nothing needs to be rotated.
|
||||
|
||||
## The Part 4 table
|
||||
|
||||
In `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` § Part 4. Short form: every Felhom secret on
|
||||
DooPlex has exposure **none**. The ones a hacked pod *could* have reached: the hub's unencrypted nightly copies (with the
|
||||
unsealed Cloudflare/tunnel/git tokens, R-975) through the NFS export or a Longhorn backup, and the operator signing keys
|
||||
through the NFS file-handle escape (documented, not exercised). Sealed hub columns need a key that is not in the copies;
|
||||
escrow blobs need each household's recovery code; the k8s API, the kubelet and the hub refuse a caller with no credential
|
||||
(measured).
|
||||
|
||||
## The paths and the evidence per path
|
||||
|
||||
| path | existed? | evidence of use (window 10-17 04:00Z → 10-23 08:04Z) |
|
||||
|---|---|---|
|
||||
| NFS export `longhorn-pvc` (`*`, `no_root_squash`, `no_subtree_check`; `secure` void in pods) | yes | **unused** — `node_nfsd_connections_total` flat at 288 for 13 days, 3902 samples, no gap; positive control: today's three probe connections moved it to 4. nfsd is TCP-only. Reads only in Longhorn's backup hours |
|
||||
| Longhorn API :9500, no login | yes | **no sign of use** — 0 backups first seen off-schedule (1-min scrape); every night wrote to the local store; no unexplained volume; the backup target is the 2025-11-29 one. Manager logs: lost (reboot) |
|
||||
| hub `hub:8080` direct | yes, but no secret without a credential | 0 requests from 10.42.0.52 in `/data/access-log` (from 10-18 10:25Z; positive control: the probe's IP is logged). 10-17 04:00Z → 10-18 10:25Z: no record |
|
||||
| k8s API, kubelet | no (401) | — |
|
||||
| SSH, Samba on DooPlex | login needed | 0 sshd lines from the pod network (control: 61 from the LAN); no Samba client file for 10.42.0.52 (positive control: the probe made one) |
|
||||
| Vaultwarden | login needed; vault encrypted | logs lost (reboot) |
|
||||
|
||||
**Evidence lost before this session started:** the 13:20Z maintenance reboot erased the pre-reboot container logs of
|
||||
longhorn-manager, longhorn-ui, the ingress, the hub, Gitea, Vaultwarden, Grafana and Prometheus. Reproduced
|
||||
independently from records that survived: Prometheus history (nfsd, Longhorn, cAdvisor, node network), the hub's own
|
||||
access log, the journal (boot -1), Samba per-client files, CoreDNS error lines, the Alertmanager silence list. The
|
||||
incident session's pre-reboot greps (10.42.0.52 in longhorn-manager, CNPG, Gitea, authentik: 0) are cited as theirs.
|
||||
|
||||
## For the homelab session
|
||||
|
||||
The seven items at the end of the audit README: the export still open to pods (and `secure` is void there); Longhorn API
|
||||
and UI with no login; an unattributed Longhorn delete of the old Plex volume at 11:24:15Z; **~1.5 TB sent out by the Plex
|
||||
pod 10-18 → 10-22** (not in INCIDENT.md) plus `hand.genddos.st` DNS lookups on 10-19; the reboot that erased the logs;
|
||||
the monitoring stack and Headlamp reachable from every pod; a `python3 main.py` listener on :8443. No HM numbers minted;
|
||||
`homelab-manifests` not touched.
|
||||
|
||||
## What this session changed
|
||||
|
||||
- **DooPlex `/etc/exports`** (operator yes, ~15:18Z): `longhorn-pvc` from `*` to `10.42.0.0/16`, `exportfs -ra`; Longhorn
|
||||
target `available: true` at 15:20:10Z. Backup `/etc/exports.bak-2026-10-23-plex2shell`. The only change on DooPlex.
|
||||
- **Probe pod** `default/cc-p2s-probe`: created 15:19:46Z, removed 15:33:44Z; before/after listings in the evidence. No
|
||||
other object created. The incident's NetworkPolicies untouched.
|
||||
- **felhom.eu:** the audit folder; R-1241 and R-1242 in `OPEN-ITEMS.md`; `13-threat-model.md` (attacker j, one boundary
|
||||
row); `STATUS.md` (verdict in „What works", the P1 in „What is broken", one „do" item; the old „Updated" paragraph moved to
|
||||
`STATUS-HISTORY.md`).
|
||||
- Nothing rotated. No secret value written anywhere (evidence scanned for secret shapes; the only 64-hex string is the
|
||||
sha256 of `/etc/exports`).
|
||||
|
||||
## Rows
|
||||
|
||||
Before 186, after 188; opened 2 (R-1241 P1, R-1242 P3), closed 0.
|
||||
|
||||
## Observations
|
||||
|
||||
- Hacked pod → hub backups + signing keys through the NFS export and the Longhorn API — FILED: R-1241.
|
||||
- Demo NAS export on the same disk with `no_subtree_check` — FILED: R-1242.
|
||||
- The hub answers pods without the geo filter — NOT-A-FINDING: the geo filter is not authentication; every no-token path refused.
|
||||
- demo-felhom's Proxmox UI and SSH reachable from any DooPlex pod — NOT-A-FINDING: the door of R-982; added as attacker j in `13` §3.
|
||||
- Pod logs erased by the reboot — NOT-A-FINDING for Felhom (the hub's access log persisted, R-1140); handed to the homelab session.
|
||||
|
||||
## Notes for the next session
|
||||
|
||||
- Teardown, three layers: machine — probe pod removed, export change kept on purpose (operator yes); host — none; hub —
|
||||
none (read-only access-log aggregation in the running pod).
|
||||
- System poster: no fact on it changed (none of the three applied).
|
||||
- `unproven.py`: not run — no capability claim changed.
|
||||
@@ -0,0 +1,106 @@
|
||||
# REPORT — the Monday release, run on Sunday 2026-10-25 (attended)
|
||||
|
||||
**Plain words first.**
|
||||
1. **Fix A and fix B are proven.** A restore whose app does not start says so and keeps both copies (D49 = A); the old
|
||||
copy goes only after 3 minutes of stable running. A `.fab` import that cannot download its version refuses and
|
||||
touches nothing.
|
||||
2. **Hub 0.170.0 is live.** Tester 1's recovery package went to the hub in one transaction; no alarm mail after.
|
||||
3. **Agent 0.165.0 is vouched** (18:38:49 CET). It runs on all four hosts; on a BYO host our SSH file is removed — one
|
||||
bundle run late (R-1285).
|
||||
4. **Controller 0.343.0 is on every box:** 4 of 4 hosts (Tester 1, demo-hp, demo-felhom, the BYO test host) and the
|
||||
scratch guest 9202. None stayed behind.
|
||||
5. **The catalog is pushed:** n8n sends no usage data, OpenGist has 256 MiB, Homepage is behind the family gate.
|
||||
|
||||
**The D47 upgrade count:** Tester 1 3 of 3 apps got an install id, demo-felhom 5 of 5, demo-hp 11 of 12 (kimai is held
|
||||
stopped since 2026-10-14 and the start-up step skips a held app — R-1286). **0 units set aside on any box.**
|
||||
|
||||
**The dated checks:** R-232 — DONE (last night's DooPlex database copy reached the restic sweep, snapshot `3973450f`,
|
||||
folder `20261024-220001`, no alarm mail since 2026-10-23 22:20Z). R-1183 — partial today (one hub snapshot file per
|
||||
slot, `07:00Z` and `13:00Z`; the hub restarted at 15:50Z for 0.170.0, so the morning log is gone); the full read stays
|
||||
due 2026-10-26.
|
||||
|
||||
## What ran, in order
|
||||
|
||||
- **Part 1, fix A (R-1270 follow-up):** one helper, „started" = running on every sample of a 45 s window, no
|
||||
`RestartCount` growth, every healthcheck `healthy`; the undo copy released after 3 more minutes (10 s samples, 30 min
|
||||
watch; only a restore begun after the watch ends it). Found live and fixed on the way: docker's `-f` template failed on
|
||||
every app without a healthcheck (now the JSON is read). Live on 9202 rc.6: an image that exits after 20 s → „még nem
|
||||
indult el", copy kept, one operator line; the original unit back → released 3 min later.
|
||||
- **Part 2, fix B (R-1279):** the D48 helper before step 1 of the import. Live: a bundle copy with a missing tag →
|
||||
refused with the D48 sentence, the app running, its volumes untouched.
|
||||
- **Part 3, hub 0.170.0 (R-1254):** asked → yes → manifest bump, hard refresh, sync of `Deployment/hub` alone,
|
||||
Synced/Healthy, digest `sha256:6478498d…`. Tester 1 ran the escrow ceremony through the page's endpoints (the new code
|
||||
stored by the operator as a CC secret); the hub logged „the new package is on the hub". No alarm mail after the
|
||||
deploy (the mailbox as the second channel; a bind mail to Tester4 arrived, the control).
|
||||
- **Part 4, agent 0.165.0 (R-1257, R-1250):** `release-agent.sh` from a clean clone; signed jobs to the three boxes
|
||||
(76/76, the appliance drop-in in force by `sshd -T`). **No BYO host existed, so one was built** on demo-hp: VM 9371, a
|
||||
plain Proxmox VE 9.2 from an answer-file ISO, Felhom installed with `--mode byo` as Tester4 (the operator deleted the
|
||||
old host and stored the retrieval passphrase as a CC secret). Our 0.164.0 drop-in planted first; after 0.165.0 still
|
||||
there; a second config bundle removed it, wrote nothing, `sshd -T` read the owner's own setting. Vouched by the
|
||||
operator.
|
||||
- **Part 5, controller 0.343.0:** rc.6/rc.7 on 9202; the set-aside alarm added and red-proved (Part 5.2); `0.343.0`
|
||||
pushed after CI 2330; per-box floors → the three boxes at 17:17Z; box checks: 0 ERROR lines, all containers up,
|
||||
https-only green; the D47 count above; on Tester 1 a unit restore and a set-aside restore — the item back, the copy
|
||||
released after 3m0s each. Asked → yes → global floor 17:53:42Z (MinAgent 0.158.1, read back in the hub log; the BYO
|
||||
host moved 18 s later); per-box floors cleared; a 30-minute watch: the hub sent 0 alarms (`global-floor-watch.txt`) and no alarm mail arrived (the mailbox as the second channel).
|
||||
- **Part 6, catalog:** Homepage's family gate tried on the drill on 9202 (stranger 302/401, household 200); asked →
|
||||
yes → `ebbf1f4` pushed; demo-hp read it 8 minutes later.
|
||||
- **Part 7, D44/D45 = all picks:** confirm lines tagged, text-follows-code changes made (old words struck through), the
|
||||
four code picks became R-1280–R-1283; R-1265 and R-1266 closed.
|
||||
- **R-1250:** released, but its live proof cannot run before the agent's next due window (Tuesday 10:00–15:59 local; all
|
||||
boxes measured on 2026-10-24) — a dated check on 2026-10-27 with the operator's yes for DooPlex's qBittorrent as peer.
|
||||
|
||||
## Versions, digests, CI by commit
|
||||
|
||||
- **Hub 0.170.0** `sha256:6478498d…`.
|
||||
- **Agent 0.165.0** binary `e9d04a47ee85222ce6c472633ecdcebe1baa88d6836848f9a7da6435a9adb7d3`, bundle
|
||||
`8dea1f34d77ba40aa44c5461c3728c2ca94bab46854a6bd8b86191f520dfe85a`; agent CI by commit: `c8d3c17` 2306 ✓, `4a1a849` 2325 ✓, `4697f8a` 2326 ✓, `6b6ba0c` 2334 ✗ (R-1290 — the registry, not the code).
|
||||
- **Controller 0.343.0** `sha256:bb3da72a054c45751e3baf6e4bad43d1561a952717c5f426d65a5aeed0413a3b` from `e2a9761`.
|
||||
CI: `7720439` 2319 ✓, `2149a13` 2321 ✓, `35232de` 2322 ✓, `1ee0353` 2324 ✓, `3a1f09d` 2327 ✓, `935e624` 2328 ✓,
|
||||
`e2a9761` 2330 ✓, `fe98377` 2333 ✓.
|
||||
- **Catalog** `ebbf1f4` 2332 ✓, `cd0cf48` (REPORT).
|
||||
- **felhom.eu** `ade357a1` 2329 ✓, `36ef8dcd` 2331 ✓, this commit (CI by hash in the final message).
|
||||
- **Installer 1.36.1** on main, NOT published (R-1284).
|
||||
|
||||
## Register
|
||||
|
||||
Before **219**, after **219**; opened **11** (R-1280–R-1290), closed **11** (R-1251, R-1252, R-1253, R-1269, R-1270,
|
||||
R-1271, R-1279, R-1257, R-1272, R-1265, R-1266). R-1254: the hub half closed in its row, the ceremony half open. R-1250:
|
||||
VERIFY with a dated check. `unproven.py --summary`: no number moved (35 of 55 not walked).
|
||||
|
||||
## Observations
|
||||
|
||||
- FILED: R-1284 — the installer's preflight printed every refusal code twice („401401"), so its 401/404/409 sentences
|
||||
never showed (R-1167's 409 sentence included); fixed on main (1.36.1, red-proved), the publish is the operator's.
|
||||
- FILED: R-1285 — BYO: our drop-in is removed one bundle run late (the old wrapper plans the run that installs the new one).
|
||||
- FILED: R-1286 — demo-hp's kimai held since 2026-10-14, no row named it.
|
||||
- FILED: R-1287 — a set-aside mails „1 of 1 apps failed to back up" (red); nothing failed.
|
||||
- FILED: R-1288 — after a set-aside restore, the next backup sets the reinstall's own unit aside too (a second copy and
|
||||
alarm).
|
||||
- FILED: R-1289 — the hub's „fleet's newest controller" can be a dev build no box runs (`0.343.0-dev.r1251a`).
|
||||
- FILED: R-1280–R-1283 — the four D44 code picks.
|
||||
- FILED: R-1290 — the agent's CI is RED since ~18:00Z: four old agent packages (0.159.0–0.160.0) were removed from the
|
||||
registry by something outside the repos, and the `published` gate still expects the newest 10 (`6b6ba0c`, job 2334).
|
||||
Not the release's code (job 2326 on the release commit was green); the keep rule is the operator's.
|
||||
- NOT-A-FINDING: the BYO host's first bundle read 71/76 (`pve:store-grant:felhom-pbs`) — the PBS storage was being added;
|
||||
76/76 two seconds later and on every run after.
|
||||
- NOT-A-FINDING: the `.fab` import status reads `"ok": true` beside its `error` — `ok` is the HTTP envelope, the job's
|
||||
result is `error` and the steps.
|
||||
- NOT-A-FINDING: 9202's alarms read „DROPPED … notifier disabled" — 9202 has no hub; the line is the positive observable
|
||||
that the event was raised (one per case).
|
||||
|
||||
## Teardown (three layers)
|
||||
|
||||
- **Machine:** every throwaway app removed through the product (9202: Vikunja, Tandoor, Homepage; Tester 1: Tandoor —
|
||||
no container, volume, `app.yaml` or set-aside left). 9202 back on the released 0.343.0, its drill mirror and my saved
|
||||
`controller.yaml` copy removed, my `0.343.0-rc.3…7` images removed (older rc images of earlier sessions left as found).
|
||||
Tester 1 keeps its new escrow package on purpose (the old recovery code no longer works; the new one is a CC secret).
|
||||
- **Host:** the BYO VM uninstalled through the product's `--uninstall` (rc 0), then VM 9371 destroyed with its disk; its
|
||||
ISO removed from demo-hp's `local`; demo-hp lists only VM 341 and its old ISO, as found. DooPlex: only the hub deploy;
|
||||
the BYO ISO work folder deleted; the session's password, session and job files shredded.
|
||||
- **Hub:** the three per-box floors cleared; the global floor 0.343.0 and the vouched manifest stay (the release). The
|
||||
host record `Tester4-f967d4` stays until the operator deletes it (session only — in STATUS); customer Tester4 is kept.
|
||||
|
||||
**Secrets:** used only through `sec`; the evidence scanned for every stored value used (0 hits) and the BYO root
|
||||
password (0 hits); one CSRF token blanked in a saved page snippet. **System poster:** no fact changed. **Instruction
|
||||
files:** none edited.
|
||||
@@ -0,0 +1,29 @@
|
||||
# REPORT — the restore and backup holes, D43 = A, and the D44/D45 picks (2026-10-24 night, unattended)
|
||||
|
||||
**Plain words first.**
|
||||
- A power cut during a restore of an app on a drive no longer brings old data back later (R-1251, the P1). Proven on
|
||||
Tester 1 with a real power-off and a drive that came back late.
|
||||
- One wrong line in an app's backup settings no longer deletes its second-drive copy (R-1252). Proven on 9202.
|
||||
- An export no longer overwrites a running restore's record (R-1253, proven on 9202); the hub saves a recovery package
|
||||
in one step (R-1254, tested, not deployed).
|
||||
- BYO hosts keep their own SSH settings (your D43 = A, R-1257). Tested in the bundle's harness.
|
||||
- The review's 32 notes have one line each with a pick: answer „all picks" or name the ones you disagree with.
|
||||
- Nothing is released. Monday: controller v0.343.0, agent v0.165.0 + your vouch, hub v0.170.0.
|
||||
|
||||
**Evidence and tables:** `documentation/audits/restore-holes-2026-10-24/README.md`; the picks:
|
||||
`documentation/audits/review-round-2-2026-10-24/D44-D45-picks.md`.
|
||||
|
||||
**Commits:** controller `bb2809d`, `062f9b4`, `518a135`, `c9b4b82`, `e1613ae`; felhom.eu `dcf4f84c` (hub) and this
|
||||
commit (documents, register, STATUS, CONTEXT, evidence); felhom-agent `c8d3c17`, `4a1a849`. CI by hash: in the final
|
||||
report.
|
||||
|
||||
**Register:** before 207, after 208; opened 1 (R-1268), closed 0. R-1251/R-1252/R-1253/R-1257 → VERIFY (fixed on main,
|
||||
waits for the release); R-1254 → NARROWED (the hub half fixed); R-1265/R-1266 carry the picks; R-1267 extended.
|
||||
|
||||
**Documents edited:** `07` (the R-1251 rule after D22's paragraph; the R-1252 rule in §8.2a; a pointer in §6.5), `05`
|
||||
(the escrow transaction), `03` and `11` §1 (D43 = A). **System poster:** no fact on it changed (none of the four).
|
||||
**Instruction files:** none edited.
|
||||
|
||||
**Machines:** Tester 1 and 9202 put back as found except leftover empty anonymous volumes of the R-1267 class (named in
|
||||
the audit). demo-hp: nothing provisioned; the catalog sweep's VM untouched. DooPlex, ep0, household boxes, the hub: not
|
||||
changed. No release, deploy, vouch, publish or catalog push.
|
||||
@@ -0,0 +1,59 @@
|
||||
# REPORT — a backup is never changed by a new install, a restore downloads first, a restore tells the truth (2026-10-25, unattended, commit only)
|
||||
|
||||
**Plain words first.**
|
||||
- **R-1269 is fixed and proven:** a reinstall no longer writes into the old backup. The old backup is set aside whole,
|
||||
listed on the restore page, and restores with its own password (Tandoor and AdventureLog on 9202: the account back).
|
||||
- **D48 download-first on every restore path that starts a definition (the unit restore and with it the second
|
||||
drive's unit and whole restore, both kept-data loads and a set-aside unit; the off-site restore; the second drive's
|
||||
file restore; the no-unit fallback).** An unpullable version is refused and the app is never stopped (proven).
|
||||
- **R-1270:** a restore whose app does not start says so — never „it worked"; the old copy is kept (proven).
|
||||
- **R-1271:** cause — the other box's off-site copy was dated in the future (its clock), and „newest wins" took it;
|
||||
now a copy more than 10 minutes in the future never counts as the newest (proven on a forged copy).
|
||||
- **One question for you:** D49 in STATUS — after a restore whose app does not start, keep the data and wait (CC's
|
||||
pick, done) or pour the old data back by itself.
|
||||
|
||||
**Safest readings taken** (also at the head of `documentation/audits/restore-truth-2026-10-25/README.md`): an install
|
||||
identity field, not the data stamps; the set-aside unit is a D25 kept copy (remove-with-backups deletes it, as it does
|
||||
other kept copies); a „partial write" means one half into another install's unit; R-1270 keeps the data and the undo
|
||||
copy (no new retention rule); D48 tells a deleted version from a busy registry by Docker's answer; 10 minutes of clock
|
||||
skew; the `.fab` import left to its own row.
|
||||
|
||||
**Architecture read:** `07` §6.3, §6.5, §6.6, §8 (D20/D22, §8.2a D25); `03` (drive identity); `09` §3 decisions 14,
|
||||
40, 53; rows R-1269–R-1271, R-1206, R-698, R-933, R-1267.
|
||||
|
||||
**Each proof, before (`e1613ae`, 9202 on 0.342.0) → after (9202 on 0.343.0-rc.2):**
|
||||
- R-1269: the kept unit's `app.yaml` rewritten (`cbfcb538…` → `e19d2610…`), restore broken → set aside byte for byte,
|
||||
restored, the account back, 0 password errors (Tandoor, AdventureLog).
|
||||
- R-1270: `ok: true` over a `starting` app → `ok: false`, `not_started`, the honest sentence, the undo copy kept.
|
||||
- D48: the first pull after the stop (code order) → an unpullable pin refused, the container's start time unchanged.
|
||||
- R-1271: the future copy chosen (sweep) → the own copy chosen; alone, the future copy marked and said.
|
||||
|
||||
**Commits.** Controller: `02daebe` (D47) CI 2311 ✓, `b680db8` (R-1270) 2312 ✓, `2d71270` (D48) 2313 ✓, `bd9f4f7`
|
||||
(R-1271) 2314 ✓, `1f9c863` (the kept list without off-site) 2315 ✓, `1f82959` (remove deletes the undo copy) 2316 ✓,
|
||||
`e8d2151` (REPORT). felhom.eu: this commit (`07` §6.5/§6.6/§8.2d/§8.2e, `09` §3 decisions 256–257, CONTEXT, register,
|
||||
STATUS, STATUS-HISTORY, the audit with its evidence and tools) — CI by hash in the final message. Test images
|
||||
`0.343.0-rc.1` (`bd9f4f7`) and `0.343.0-rc.2` (`1f9c863`) in the registry; nothing released.
|
||||
|
||||
**Teardown.** Machine: 9202 only; my throwaway apps removed through the product; two empty folders and one undo copy of
|
||||
mine removed by hand after checking; my rc images removed from its Docker; back on 0.342.0; the apps found running still
|
||||
run. Host: the session's 9202 password and session files shredded. Hub: provisioned nothing. ep0: nothing.
|
||||
|
||||
**Register:** before 217, after 219; opened 2 (R-1278, R-1279), closed 0; R-1269, R-1270, R-1271 → VERIFY (fixed on
|
||||
main, wait for the release); notes on R-1206 (cross-reference) and R-1243 (its dated check done, read-only); R-232's
|
||||
dated check moved one day (this brief: no DooPlex work).
|
||||
|
||||
**Observations:**
|
||||
- FILED: R-1278 (a volume app's reinstall does not offer its kept backup).
|
||||
- FILED: R-1279 (the `.fab` import removes the app's volumes before its pull — D48 does not cover it).
|
||||
- FIXED without a row (found live, red-proved): the kept-copies list was hidden on a box without off-site (`1f9c863`);
|
||||
a kept undo copy outlived remove-with-backups (`1f82959`).
|
||||
- NOT-A-FINDING: Immich's 404 after the sweep's load — the copy brought its own address (`photos.`), as every unit
|
||||
restore does; the app was healthy there.
|
||||
- NOT-A-FINDING: „immich: definition REFUSED by the allow-list" on 9202 — present before this run's image; an earlier
|
||||
session's Immich, not this change.
|
||||
- NOT-A-FINDING: the D48 refusal reads „Visszaállítás sikertelen: …nem lehet letölteni… Az alkalmazáshoz nem
|
||||
nyúltunk" — the page's failure frame around an honest sentence; true as written.
|
||||
|
||||
**Secrets:** used only through `sec`; the evidence scanned for the box password and the session tokens (the CSRF token
|
||||
in two saved page snippets was the positive control and is blanked); units compared by sha256 only. **System poster:**
|
||||
no fact changed. **Instruction files:** none edited.
|
||||
@@ -0,0 +1,33 @@
|
||||
# REPORT — review round 2 (2026-10-24, unattended): the documents against the code, data loss and security
|
||||
|
||||
**Plain words first.**
|
||||
- The 17 design documents were checked against the code: 493 claims, 121 out of date. About 130 places are corrected or
|
||||
marked with a dated note; no rule, fence or promise was changed.
|
||||
- About 30 things the code does that no document named are now written down for you to confirm (D44), and seven places
|
||||
where a document's ruling or promise says something the code does not do are waiting for you (D45).
|
||||
- The worst data-loss finding: a power cut during a restore of an app on a drive can later pour old data over the
|
||||
household's new data (R-1251, P1). Proven with the real recovery code; it needs a fix in an attended run.
|
||||
- Also proven, on the scratch box: one wrong line in an app's backup settings deletes that app's second-drive copy
|
||||
(R-1252, P2).
|
||||
- The worst security finding: the „root logs in by key only" SSH rule also reaches a host the household owns (BYO),
|
||||
where we promised to change nothing (R-1257, D43). Nothing serious was found in the hub's or the controller's new routes.
|
||||
|
||||
**Evidence and the full tables:** `documentation/audits/review-round-2-2026-10-24/README.md`; every document edit with
|
||||
its `file:line` in `EDITS.md`; the reviewers' candidates in `reviewers/`; the two proofs in `proofs/`.
|
||||
|
||||
**Commits (CI read by hash):** controller `b01bb4b` (two small fixes, red-proved; unreleased) — CI 2294 / #480 success;
|
||||
controller `2449223` (REUSE.md) — 2298 / #481 success; felhom-agent `f5c3d77` — 2296 / #261 success and catalog
|
||||
`227a544` — 2297 / #256 success (the rule-file alignment); felhom.eu `77015997` (documents batch 1) — 2295 / #1253
|
||||
success; felhom.eu `876d9236` (batch 2, the register, STATUS, CONTEXT) — 2299 / #1254 success.
|
||||
|
||||
**Register:** before 190, after 207; opened 17 (R-1251–R-1267), closed 0.
|
||||
|
||||
**Instruction files edited (facts only, named here):** `.claude/rules/unprompted-work.md` in felhom-agent, the catalog
|
||||
and the workspace root — the missing „A version number is pushed once" paragraph (before 95 lines, md5 3b071ad2; after
|
||||
byte-identical to the other two, b1891bd4); `felhom.eu/.claude/rules/live-probes.md:41` — `/opt/felhom/stacks/<name>` →
|
||||
`/opt/docker/stacks/<name>/app.yaml` (the real path, and the folder stays after a clean remove).
|
||||
|
||||
**System poster:** the facts file and the html changed together (text only, the hub-DB push times); no new drawing.
|
||||
|
||||
**Machines:** 9202 used for the R-1252 proof and put back (one difference: Immich's new empty volume, R-1267). DooPlex,
|
||||
ep0, household boxes and the hub: not changed. No release, deploy, vouch or publish.
|
||||
@@ -0,0 +1,17 @@
|
||||
# REPORT — D39 = B: the security-fix lane (2026-10-23, attended)
|
||||
|
||||
**Plain words first.** An app with a public security hole can now get its fix at night on every box, after one test
|
||||
on the test box. Controller 0.341.0 does it and is the floor since 13:24 (your yes). Older controllers skip such a fix
|
||||
at night and take it when a person presses Update — both seen on the test box. Nothing needs you.
|
||||
|
||||
- Design: `09-update-architecture.md` §3 decision 255 (+ a dated note on decision 13). Capability map B23.
|
||||
- Catalog `3898593` (+ a box-walk address fix after it); controller `63dcccf` = 0.341.0; drill commits `8c1d908` …
|
||||
`17e0da0` (the drill only).
|
||||
- Evidence and the full table: `documentation/audits/security-lane-2026-10-23/README.md`.
|
||||
- Register 186 → 186: closed R-1239 (D39 = B); opened R-1240 (9202's 34 saved configs with the catalog token); R-1238
|
||||
stays open, its text now says the next Plex fix can use the lane.
|
||||
- `STATUS.md`: D39 out of the decide table, one „what works" line, the „Updated" paragraph replaced (the old one is in
|
||||
`STATUS-HISTORY.md`). `CONTEXT.md`: the rulings. System poster: no fact changed (none of the three actions).
|
||||
- Instruction / fact files edited: `app-catalog-felhom.eu/CLAUDE.md` (the lane, added); `operations/nodes.md` line 221
|
||||
(9202 „same dashboard password" → its own password, measured) and line 46 (Tester 1 `.101` → DHCP, `.148` today).
|
||||
- No `--no-verify` in this run.
|
||||
@@ -0,0 +1,129 @@
|
||||
# REPORT — weekend night 2026-10-23: the hub's last plain tokens locked; the upload and CPU watch (spike, design, build)
|
||||
|
||||
**Safest readings taken (unattended), stated first:**
|
||||
- **A test image is not a release.** The upload watch was proven on scratch 9202 with `felhom-controller:0.342.0-rc.1`,
|
||||
set by hand in 9202's `/etc/felhom-controller-image` (the shape of today's security-lane rc.1). An `-rc` tag is never
|
||||
offered to a household's update button. No floor was touched; only 9202 ran it, and 9202 is back on 0.341.0.
|
||||
- **9202's line was seeded by hand** (one 40 Mbit/s record): its off-site backup is off by design, so it can never learn a
|
||||
line itself. The learning path is pinned by unit tests and by tonight's one-second read of the real pushes on the
|
||||
three test boxes (read only).
|
||||
- **The hub 0.168.0 image was built** (the brief asked for it) **and not deployed.** The four new hub event types are in
|
||||
hub 0.169.0, committed only.
|
||||
- **Per-app trigger instead of a whole-box trigger.** The brief said „the box's upload above X % of the line". Inside its
|
||||
container the controller cannot tell LAN from internet bytes, nor see the box's total; it can see each app's exact
|
||||
bytes, and the per-app view is what the explanation compares. So the trigger is per app (§6.2 of the spike).
|
||||
- **The first day:** apps the box can ask are judged at once on the 5 Mbit/s floor (a mismatch is evidence whatever the
|
||||
line); apps it cannot ask are judged only after ~20 h of history. The brief's „first day: no alarm, only learning" holds
|
||||
for those apps; for the others a mismatch on day one is still an alarm.
|
||||
|
||||
## Five lines
|
||||
|
||||
1. **Tokens locked:** the Cloudflare tunnel token, the Cloudflare API token and the git token are sealed in **hub 0.168.0**
|
||||
— built, CI green, **not deployed** (you deploy it on Monday).
|
||||
2. **Backup copies:** two options — encrypt the hub's Longhorn disk, or take it out of the Longhorn backup because the
|
||||
hub's own copy to ep0 is already encrypted at rest and in transit (measured). **My pick: take it out** (D40).
|
||||
3. **The line:** measured from the box's own nightly off-site upload (one-second samples, best 10 seconds) — a lower
|
||||
bound; most nights the upload is too small to say anything, so a real test is your decision (R-1247).
|
||||
4. **Apps that explain their traffic:** qBittorrent and Transmission (their own byte counters, within 1–7 %), Plex (its
|
||||
viewers and transcodes). Jellyfin needs a key the household makes; Nextcloud and Immich give no number.
|
||||
5. **The fake hacked app was caught: yes** — a flood from inside qBittorrent, hidden behind a real seed: one alarm after
|
||||
15 minutes, one clear 15 minutes after it stopped. A CPU burner in Plex: the same. A real seed: no alarm.
|
||||
|
||||
## The spike's numbers
|
||||
|
||||
`documentation/audits/SPIKE-upload-watch-2026-10-23.md` (evidence beside it). Short form:
|
||||
|
||||
- **Normal levels** (three boxes, 19–31 days of one-minute samples): no app's own upload p95 reached 6 Mbit/s; the
|
||||
longest run at ≥ 5 Mbit/s was **3 minutes**, at ≥ 1 Mbit/s 9 minutes. CPU: the longest run at ≥ 150 % of a core was
|
||||
**7 minutes**, at ≥ 80 % 13 minutes. DooPlex's hacked Plex: ~44 Mbit/s for hours, ~10 cores.
|
||||
- **Line speed:** at a one-minute sample the push exceeded 2 Mbit/s on 3 of 81 box-nights (the push moves a few MB);
|
||||
the box keeps no byte count of it. Tonight's one-second read: §Night data below. No active test was run (ep0 changes
|
||||
forbidden; a public test's terms not read).
|
||||
- **Apps:** qBittorrent counted 131.1 Mbit/s while its container sent 134.6–140.5 (×1.03–1.07); Transmission ×1.001–1.02;
|
||||
the limit API works and persists (KiB/s rounding); Plex answers its session lists (with the token it already has when
|
||||
claimed); Jellyfin 401 without a household-made key.
|
||||
- **A hard cap** (`tc` from a helper in the app's network namespace) works in an unprivileged guest: a flood held at
|
||||
20 Mbit/s, the app kept answering. Not built (R-1248).
|
||||
|
||||
## The household's line (final text)
|
||||
|
||||
- HU: „A qBittorrent feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
|
||||
- EN: "qBittorrent uploads at most N Mbit/s, so the family internet stays fast. You can change this in qBittorrent."
|
||||
- The system settings page: „Feltöltési sebesség: legalább N Mbit/s (az éjszakai távoli mentésből mérve)" /
|
||||
"Upload speed: at least N Mbit/s (measured from the nightly off-site backup)"; „még mérjük …" until known.
|
||||
|
||||
## Morning list (Monday)
|
||||
|
||||
1. Deploy hub 0.168.0 (the last plain tokens locked). Then read its start line `config tokens sealed at rest (8 …)`.
|
||||
2. D40: the hub's plain Longhorn copies — stop that backup (my pick) or encrypt the disk; and whether to delete the two
|
||||
plain copies already there.
|
||||
3. D41: qBittorrent's default limit — 50 % of the measured line (my pick). D42: who is told — the family only about the
|
||||
limit, the hacked-app alarms only you (my pick).
|
||||
4. Release the upload watch when you say yes (hub 0.169.0 build + deploy, then controller 0.342.0 through the normal path).
|
||||
5. For the homelab session (list only, no HM numbers): an upload-saturation alert on DooPlex's `eno1` (the line is
|
||||
~100 Mbit/s; the flood peaked at 905 Mbit/s).
|
||||
|
||||
## Commits, images, CI (by commit)
|
||||
|
||||
| Repo | Commit | What | CI |
|
||||
|---|---|---|---|
|
||||
| felhom.eu | `5277d859` | hub 0.168.0: the tokens sealed; R-1243; STATUS D40 | run 2269 **success** |
|
||||
| felhom.eu | `af5c005a` | hub 0.169.0: the four events (allowed, operator-only, mail line) | run 2271 **success** |
|
||||
| felhom.eu | `a91d0b24` | spike, design (`08` §9), live proofs, rows R-1244..R-1248, STATUS | run 2272 **success** |
|
||||
| felhom-controller | `a9c1997f` | v0.342.0 (unreleased): the upload and CPU watch | run 2270 **success** |
|
||||
| felhom-controller | `60e487d` | CONTEXT + REPORT | run 2273 **success** |
|
||||
|
||||
Images: `felhom-hub:0.168.0` `sha256:3bfdecaae527d4762059600cdfaad83daf4e4b5b20661fc47acf128180d87cb3` (from `5277d859`,
|
||||
not deployed); `felhom-controller:0.342.0-rc.1` (from `a9c1997f`, scratch 9202 only, removed from 9202 after).
|
||||
|
||||
**Teardown, three layers:** machine — 9202: the four throwaway apps removed through the product (containers and volumes
|
||||
gone), my test files, probes and the `alpine:3.20` helper image removed, back on 0.341.0, the seeded record removed;
|
||||
host — demo-hp: the receiver LXC 9471 destroyed with its disk; DooPlex: my private copy of the hub snapshot shredded,
|
||||
the 9202 password copy shredded; hub — none. ep0: read only (the hub-copy snapshot list). No household box touched.
|
||||
|
||||
## Rows
|
||||
|
||||
**Before 188, after 194; opened 6 (R-1243 … R-1248), closed 0.** R-975 stays open (WAITING-ON-OPERATOR: deploy 0.168.0).
|
||||
|
||||
## Observations
|
||||
|
||||
- The hub's Cloudflare/tunnel/git tokens unsealed in `config_json` — FIXED in hub 0.168.0 (R-975, not yet deployed).
|
||||
- The hub's Longhorn copies are plain and sit in the open folder — FILED: R-1243.
|
||||
- The upload watch is not released — FILED: R-1244; its limit size — FILED: R-1245; who is told — FILED: R-1246.
|
||||
- The line is only a lower bound — FILED: R-1247. An app's API can lie — FILED: R-1248.
|
||||
- `appliance_registrations.pairing_code` is plain — NOT-A-FINDING: it is shown on the box's screen and is one of two
|
||||
factors; the other is hashed.
|
||||
- R-879's sealing (0.138.0) ran no VACUUM, so old plaintext of those columns could sit in freed blocks of the live
|
||||
`hub.db` — NOT-A-FINDING: 0.168.0's one-time VACUUM rewrites the file on its first start.
|
||||
- The hub CHANGELOG's 0.167.0 header still said „NOT built" though it is deployed — fixed without a row (small).
|
||||
- The shared memory index went over its size ceiling twice in one day (another session's lines) and blocked the push
|
||||
gates — fixed without a row (shortened, no link lost); it sits 14 bytes under the ceiling now.
|
||||
- The watch's very first minute after a start cannot reach apps by name (network joined seconds later) — NOT-A-FINDING:
|
||||
that minute only sets baselines.
|
||||
- The media wiring's generated override file stays after an app's removal — NOT-A-FINDING: rewritten before every start.
|
||||
- `docker stats` text in the stored metrics has three significant digits — NOT-A-FINDING for the watch (it reads the
|
||||
Engine API); the charts are display only.
|
||||
- `felhom-samba` runs on the host network and has no counter — NOT-A-FINDING: infrastructure, not judged.
|
||||
- Transmission's rate under its limit was not observed (no peer contact) — NOT-A-FINDING: the limit API was read back;
|
||||
qBittorrent's was observed.
|
||||
- A dated check came due on the push (R-1214, Tester 1's night copy): read read-only — the guest's memory „full" stall
|
||||
was 444 µs in total over 11 h, 0 OOM kills; the copy was small (0.24 GB) and the capped container is not shown — the
|
||||
row stays WATCHING, the due line removed.
|
||||
- The night's line floor is low (10 Mbit/s on demo-hp, one second at 70) — FILED in R-1245's text (D41), no new row.
|
||||
|
||||
## Instruction files, poster
|
||||
|
||||
No instruction file edited. System poster: no fact on it changed (nothing deployed) — none of the three applied.
|
||||
|
||||
## Night data
|
||||
|
||||
Read 02:08–02:30Z, once a second, on the three test boxes (read only; spike §2.1):
|
||||
|
||||
- **demo-hp: counts.** Best 10 seconds 12.6 MB = **10.0 Mbit/s**. One second reached 70 Mbit/s.
|
||||
- **demo-felhom: too small** (best 10 seconds 3.2 MB = 2.5 Mbit/s).
|
||||
- **Tester 1: too small** (3.5 MB = 2.7 Mbit/s).
|
||||
|
||||
**What it means:** the nightly push is a burst of a few seconds. So the line it gives is a low floor. On demo-hp,
|
||||
50 % of that floor is about 5 Mbit/s for qBittorrent — probably much less than the real line. The floor rises only when
|
||||
a bigger night comes (the box keeps the best of 14 nights). Added to the D41 decision row (R-1245): if you want a
|
||||
looser limit before a real line test exists (R-1247), pick a higher share or „none".
|
||||
@@ -96,7 +96,7 @@
|
||||
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
|
||||
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
|
||||
| `handleHostEscrowGet` + `escrowSelfServiceRetrieval` | hub/internal/api/handler.go | `GET /hosts/{id}/escrow` | The box-authenticated MIRROR of the escrow PUT — self-scoped by the per-host key. **`escrowSelfServiceRetrieval` is THE single decision point** for whether a box may read its own blob without operator-armed recovery mode (§8.2 vs §8.3, R-199): flip it, do not scatter the condition. **Never merge this with `dr.go`'s `handleReEnroll`/`handleGetRestoreDirective`** — those rotate the host API key and serve the K-escrow and directive too, and keep their recovery-mode gate (pinned by `TestEscrowGet_OperatorDRPathUnchanged`). Every successful retrieval MUST raise `escrow_blob_served` before the bytes leave; that audit row is the mitigation the trade rests on. |
|
||||
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts** — `blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** `SaveHostDRBundle` writes `identity_blob` AFTER `SaveHostEscrow` returns, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
|
||||
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow` / `SaveHostEscrowWithIdentity`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts** — `blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** since R-1254 (unreleased, next hub v0.170.0) `SaveHostEscrowWithIdentity` writes the key, its hash and `identity_blob` in ONE transaction, with the demote FIRST inside it, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. `SaveHostEscrow` is that function with no identity blob; `SaveHostDRBundle` (a second statement) is no longer called by the escrow PUT. Pinned by `TestR1254_*`. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
|
||||
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
|
||||
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
|
||||
| `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |
|
||||
|
||||
@@ -2,26 +2,28 @@
|
||||
|
||||
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
|
||||
|
||||
**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night PASSED (no failed check at night, every night job passed, nothing lost). Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35).
|
||||
**Updated 2026-10-25 (evening, attended release).** Every box runs the new controller: a restore that does not start says so and keeps both copies (D49), a reinstall never touches the old backup, a restore downloads first. The hub stores the recovery package in one step; the new agent is vouched; the catalog has three fixes.
|
||||
|
||||
## What works
|
||||
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
|
||||
- A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4).
|
||||
- 65 catalog apps install, update one tested step at a time, and a failed update undoes itself (map B1, B3, B4, B6).
|
||||
- Backups: database dumps, a second-drive copy, an off-site copy and whole-box copies; restore tests run by themselves (map C1–C4, A17, A18).
|
||||
- A restore brings an app back with its data and its database password; a failed or cut restore puts the app back as it was — off-site live, local on the test image (map C5, D6, D16).
|
||||
- A restore brings an app back with its data and its database password; a failed or cut restore puts the app back as it was; a restore that does not start says so and keeps both copies (map C5, D6, D7, D16).
|
||||
- After a power cut the box starts by itself, cleans up a half-written backup, and Immich repairs broken photos (map C10, F12).
|
||||
- A second drive is offered as the backup target; media apps see every drive; a one-disk box gets its own data area (map D1, D13, D15).
|
||||
- The three test boxes answer only over https; the hub's key opens nothing on a box; password guesses are counted (map E12, E13, E17).
|
||||
- The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15).
|
||||
- A hacked app that uploads or computes more than it explains alarms only you; each box tests its upload weekly; qBittorrent gets half (map F13).
|
||||
- An urgent security fix reaches the boxes at night after one test on the test box; the box backs up first and undoes a failed update (map B23).
|
||||
- The hub mails alarms and recoveries and keeps a weekly whole-box copy on ep0; CC uses its own limited token (map F1, F11, G15).
|
||||
|
||||
## What is broken
|
||||
- **P1:** about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
|
||||
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862).
|
||||
- DooPlex: the failed disk is still in the box, its alarms silenced (R-934); something sets the shared checkout to „bare" (R-935).
|
||||
- The scratch box cannot use agent features (R-1083).
|
||||
- **P1:** any hacked app on DooPlex can read the hub's nightly backups, and maybe the signing keys (R-1241, homelab session); about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
|
||||
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). The scratch box cannot use agent features (R-1083).
|
||||
- DooPlex: something sets the shared checkout to „bare" (R-935).
|
||||
|
||||
## What we work on
|
||||
- Dated checks, run by CC. 2026-10-23: no app stop when the host is busy (R-1173). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts).
|
||||
- Dated checks, run by CC. 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the qBittorrent line test on a seeding box, with DooPlex's qBittorrent as the peer (R-1250, your yes); the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes).
|
||||
|
||||
## Waiting on you — decide
|
||||
|
||||
@@ -29,11 +31,8 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
|
||||
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
||||
|---|---|---|---|---|---|---|
|
||||
| D36 | R-1225, R-1226 | Two website sentences before the publish: „8 GB runs the basics; the photo app needs 16 GB” and „what you open from outside the home passes Cloudflare's network, readable there; the backups leave only encrypted”? | **Yes, both** (the patch's own aim is the whole truth) | — | the patch publishes with both promises a little too strong | — |
|
||||
| D35 | R-1228 | After the release week, serve the website and the installer from Cloudflare, like the ISO? | **Yes, later** (no new installs stop when DooPlex is down) | keep them home: after a DooPlex loss, no install until it is back | the same | — |
|
||||
| D34 | R-232 | Copy the hub database to ep0 every 6 hours instead of every night? | **Yes** (0 €, about 30 minutes, one DooPlex change) | keep nightly: up to a day of hub changes lost, e.g. a new box's recovery package | the same | — |
|
||||
| D33 | R-1227 | Copy the released images and agent packages to ep0 every night? | **Yes** (0 €, about 2 hours of CC, a few GB on ep0) | a timed rebuild drill instead: about 3 hours once, and rebuilt images are not the same | after a DooPlex loss no box updates, reinstalls or installs until everything is rebuilt | — |
|
||||
| D32 | R-923, R-924 | Print the break-glass sheet now (eight keys and logins + ep0's fingerprint) and keep it outside the home? | **Yes** (30 minutes; nothing else opens the off-site copies after a loss) | an encrypted USB stick outside the home: ~10 €, 1 hour, one more passphrase | after a DooPlex loss no off-site copy opens | — |
|
||||
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
|
||||
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
|
||||
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
|
||||
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
|
||||
|
||||
@@ -42,23 +41,21 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
**Still needed? (yes/no)** — CC could not check these. Answer yes or no per line.
|
||||
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). Since 2026-10-16.
|
||||
2. The N100 sheet, about 30 minutes: Plex, the admins in Prowlarr, Sonarr and Radarr, your download sources (`documentation/audits/n100-lived-in-2026-10-15/SHEET.md`). Since 2026-10-15.
|
||||
3. Look at these pages on demo-hp when you have time: Calibre-Web („Adatok és mentés"), the file manager, AdventureLog, Plex, the app page, the Plex and Sonarr guides („Útmutató"). Since 2026-10-16.
|
||||
3. Look at these demo-hp pages when you have time: Calibre-Web, the file manager, AdventureLog, Plex, the app page, the Plex and Sonarr guides. Since 2026-10-16.
|
||||
4. Remove the test server „plex-k6" from your Plex account (plex.tv → Settings → Authorized Devices). Since 2026-10-15.
|
||||
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). Since 2026-10-14.
|
||||
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
|
||||
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
|
||||
|
||||
**Morning, 2026-10-23 — in this order.**
|
||||
1. Hold or go: CC says go — the test night passed on a fresh 8 GB box (no failed check at night, every night job passed, nothing lost; one gap: no whole-box copy to ep0, R-1232). One question from the review: D31 as built also covers an install that got the whole-box password but no recovery package — its password is rotated without your press (R-1205). CC: go. If not: revert one hub commit before item 4.
|
||||
2. Release controller 0.340.0 (main after tonight; tested as rc.8 on the scratch box, rc.6 on the test box): the safe night copy, the catch-up fix, R-49, R-1187, R-1189, R-1214. If not: these wait.
|
||||
3. Release the agent (the tunnel repair, its tunnel field, the root-SSH bundle) and the installer tag with the root-SSH removal (R-1204, R-982); then vouch. If not: a clock-ahead box stays cut off; root keeps its password on the LAN.
|
||||
4. Build and deploy hub 0.167.0 (D31, the tunnel alarm and its clearing). If not: a dead tunnel stays silent.
|
||||
5. Publish the website patch (D3, D5), with D36's two sentences if yes. If not: the old wording stays.
|
||||
6. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef (tonight's test box). If not: two dead entries alarm.
|
||||
7. The DooPlex hour: D32–D35 in the table. If not: a DooPlex loss stays as the survey describes.
|
||||
**Day, 2026-10-23 — your do list.**
|
||||
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
|
||||
2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-f967d4 (the BYO test host, removed 2026-10-25). If not: dead entries alarm.
|
||||
3. Give the homelab session R-1241, R-1242, the lists at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` and `documentation/audits/dooplex-alarms-2026-10-24/README.md`, and an upload alarm on DooPlex's `eno1` (~100 Mbit/s line; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups.
|
||||
**The rest**
|
||||
- Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12.
|
||||
- System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy.
|
||||
- Tell CC what removed four old agent packages from Gitea on 2026-10-25 (a clean-up rule?): the agent's CI is red until the gate and the rule agree (R-1290).
|
||||
- Publish installer 1.36.1 (tag + webpage sync): a wrong passphrase then reads „passphrase REJECTED", not a doubled code (R-1284).
|
||||
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
|
||||
- Two leftovers on DooPlex: the Facebook `first-post/` folder in `leftovers/2026-10-14/` (keep, commit or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). If not: they stay. Since 2026-10-14.
|
||||
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784); print the break-glass sheet and the signing keys (R-923, R-924).
|
||||
- Later (yours, not urgent): R-1229 (ArgoCD's sync settings into git), R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
|
||||
- Two DooPlex leftovers: `leftovers/2026-10-14/first-post/` (keep or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). Since 2026-10-14.
|
||||
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784).
|
||||
- Later (yours, not urgent): R-1250 (a seeding qBittorrent is never tested), R-1248 (a hard upload cap per app), R-1229 (ArgoCD's sync settings into git), R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
|
||||
@@ -80,17 +80,17 @@
|
||||
| <a id="a4"></a>[A4](capability-map-detail/A.md#a4) | Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; universal secret-free / operator-bind mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE** | N100 (2026-07-18) and HP t740 (2026-07-21) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/SPIKE-baremetal-iso-2026-07-16.md` (2026-07-16) |
|
||||
| <a id="a5"></a>[A5](capability-map-detail/A.md#a5) | Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`<iso>.rootpw.txt`) | scripts v1.24.0 (`felhom-bootstrap.sh`, `build-felhom-iso.sh`) | **PROVEN-LIVE** | nested drill; metal proof pending | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` (2026-07-22) |
|
||||
| <a id="a6"></a>[A6](capability-map-detail/A.md#a6) | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** | drill VM; no non-operator human yet (R-3) | `audits/DRILL-day0-vm-2026-07-12.md` (2026-07-12) |
|
||||
| <a id="a7"></a>[A7](capability-map-detail/A.md#a7) | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** | customer-zero, on metal | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
||||
| <a id="a8"></a>[A8](capability-map-detail/A.md#a8) | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** | drill VM, endpoint-exact; browser wizard first fired 2026-07-18 | `audits/SPIKE-controller-escrow-2026-07-13.md` (2026-07-13); `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
||||
| <a id="a7"></a>[A7](capability-map-detail/A.md#a7) | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** | customer-zero, on metal; **re-check (2026-10-24):** mechanism changed after its evidence — f90e2229 (bind mail W3–W5) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
||||
| <a id="a8"></a>[A8](capability-map-detail/A.md#a8) | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** | drill VM, endpoint-exact; browser wizard first fired 2026-07-18; **re-check (2026-10-24):** mechanism changed after its evidence — agent c11e3e2 + controller ba4a6f0 (typed-back words, R-1151) | `audits/SPIKE-controller-escrow-2026-07-13.md` (2026-07-13); `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
||||
| <a id="a9"></a>[A9](capability-map-detail/A.md#a9) | One name per secret on every surface — „Beállító kód" (the setup code) and „Tulajdonosi jelmondat" (the owner passphrase) — and the re-enrol mail points at a page a rebuilt box shows | controller v0.211.0, hub v0.104.0 + v0.105.0 (R-295, R-323) | **IMPLEMENTED** | written BUILT: source-read; no customer has typed the passphrase | R-295; R-323 |
|
||||
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
|
||||
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
||||
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints; **re-check (2026-10-24):** mechanism changed after its evidence — hub 9688e76e (sealed package back to a rebuilt box, R-1176) | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
|
||||
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only; **re-check (2026-10-24):** mechanism changed after its evidence — agent ae69ad4 (D30 WireGuard re-key) | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
||||
| <a id="a12"></a>[A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES — the rebuilt Tester4 box (public ISO) got the household's sealed package from the hub, asked for the recovery code, opened the history; Nextcloud came back with 3 031 of 3 033 manifest files byte-identical (the 2 never reached the old box) and the next off-site runs appended (63 → 64). Earlier the same day it FAILED (R-1176, now closed); the failure paths are not claimed (R-379, R-380) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) |
|
||||
| <a id="a13"></a>[A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES with ONE operator act by design (D19: delete the dead host) — install, bind, claim, the recovery offer, a wrong code refused, the right code, „Teljes visszaállítás", all through the household's own pages; the new-link page now tells a household whose old box is still linked to write to Felhom (R-1203). Earlier the same day it stopped at the rebuilt dashboard (R-1176) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) |
|
||||
| <a id="a14"></a>[A14](capability-map-detail/A.md#a14) | A random night of household actions while random things go wrong — twelve rounds, unattended | controller v0.245.0, agent v0.131.0, hub v0.116.0, golden 0.245.0, ISO 1.28.0 | **PROVEN-LIVE** | nested box, twelve chaos rounds | `audits/DRILL-chaos-night-2026-09-17.md` (2026-09-17) |
|
||||
| <a id="a15"></a>[A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.336.0, agent v0.161.0, hub v0.162.0, golden 0.331.0, installer 1.35.0, ISO 1.29.0 | **PARTIAL** | 2026-10-20: download → install → bind → claim → 8 apps → use → backup → remove → every restore path → power cut → mistyped codes all worked; the 8 GB box FROZE twice in the first afternoon and only a plug pull brought it back (R-1192), and the owner passphrase needed the operator | `audits/rehearsal-2026-10-20/journal.md` (2026-10-20); `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) |
|
||||
| <a id="a16"></a>[A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) |
|
||||
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
|
||||
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp; **re-check (2026-10-24):** mechanism changed after its evidence — agent 1b6e316 (local keep-daily, R-1149) | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
|
||||
| <a id="a18"></a>[A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites per-repo CHANGELOGs and the spec `backlog/SPEC-r85-phase4-5-2026-07-26.md` — a spec, not a run record |
|
||||
| <a id="a19"></a>[A19](capability-map-detail/A.md#a19) | A restore-test can never fill the box's disk: it sizes the restore first (uncompressed), keeps off the tested guest's pool when it can, refuses what does not fit, and retries a failed clean-up on a timer | agent v0.133.0 (RELEASED, NOT DELIVERED), hub v0.124.0 | **IMPLEMENTED** | written BUILT: the refusal proven on demo-hp 2026-09-24; the full-test path not live | `audits/r672-2026-09-24/` (2026-09-24); R-672 |
|
||||
| <a id="a20"></a>[A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0; purge-first hub v0.142.0 | **PROVEN-LIVE** | two real firings (2026-07-18); third 2026-10-22 on a real customer (Tester-2) with the R-32 purge-first leg — all legs ok, others untouched | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/tester2-reset-2026-10-22/` |
|
||||
@@ -102,17 +102,17 @@
|
||||
|
||||
| ID | Capability | Component | Status | Scope | Evidence |
|
||||
|---|---|---|---|---|---|
|
||||
| <a id="b1"></a>[B1](capability-map-detail/B.md#b1) | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/RERUN-p1p3-2026-07-06.md` (2026-07-06) |
|
||||
| <a id="b2"></a>[B2](capability-map-detail/B.md#b2) | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | the actions; remove-with-data re-proven 2026-09-13 | `audits/R442-2026-09-13/` (2026-09-13); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
||||
| <a id="b1"></a>[B1](capability-map-detail/B.md#b1) | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (65 apps, images pinned) | **PROVEN-LIVE** | **re-checked 2026-10-24** on controller 0.342.0 (after 6d90374 + 2fe6341): all 65 apps through the deploy request on a fresh box — 62 installed and answered, Wanderer installed but cannot run without DNS (R-739), 2 not offered by lifecycle; the memory guard did not fire (one app at a time) | `audits/catalog-sweep-2026-10-24/README.md` (2026-10-24); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/RERUN-p1p3-2026-07-06.md` (2026-07-06) |
|
||||
| <a id="b2"></a>[B2](capability-map-detail/B.md#b2) | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | the actions; remove-with-data re-proven 2026-09-13; **re-check (2026-10-24):** mechanism changed after its evidence — 8ae9178 (removal deletes labelled volumes, R-1117) | `audits/R442-2026-09-13/` (2026-09-13); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
||||
| <a id="b3"></a>[B3](capability-map-detail/B.md#b3) | Update is GUARDED: it refuses without a restorable backup, backs up first when the copy is stale, and HOLDS an app that does not come up — on ANY backup tier, and the release itself arrives by the managed floor | controller v0.237.0 + v0.238.0 + v0.238.1 + v0.239.0, hub v0.112.0 | **PROVEN-LIVE** | — | `audits/slice4-2026-09-13/` (2026-09-13); `audits/rulings-r472-r475-2026-09-13/` (2026-09-13) |
|
||||
| <a id="b4"></a>[B4](capability-map-detail/B.md#b4) | A failed update is UNDONE by the box itself — the previous version back with its data from seconds before the update; the app is held only if that undo fails too | controller v0.263.2 (the undo), v0.264.0 + hub v0.120.0 (the mail) | **PROVEN-LIVE** | scratch 9202 | `audits/undo-live-2026-09-23/README.md` (2026-09-23) |
|
||||
| <a id="b5"></a>[B5](capability-map-detail/B.md#b5) | The catalog holds only TESTED steps — an image move without a proven test record (bench + box, digests the registry still serves) is refused at push time | catalog `6db08a5` (test-record gate, `ladder.py`) | **PROVEN-LIVE** | written PROVEN: the catalog gate against the live registry | `audits/DRILL-night-2026-09-23.md` (2026-09-23) |
|
||||
| <a id="b6"></a>[B6](capability-map-detail/B.md#b6) | A box behind climbs ONE tested step per press, each with its own definition (`09` §3 decision 14) | controller v0.268.0 + catalog `5ed599c` | **PROVEN-LIVE** | — | `audits/ladder-2026-09-24/partD/10-romm-two-steps.json` (2026-09-24) |
|
||||
| <a id="b7"></a>[B7](capability-map-detail/B.md#b7) | A file app comes back WHOLE from the second drive — its files by four rules (never delete, never overwrite a newer file, bring back what is missing, keep an older replaced file beside it), then its settings and database (`09` §3 decision 26) | controller v0.269.0 (`backup/tier2_whole.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A1/21-readback.json` (2026-09-24) |
|
||||
| <a id="b7"></a>[B7](capability-map-detail/B.md#b7) | A file app comes back WHOLE from the second drive — its files by four rules (never delete, never overwrite a newer file, bring back what is missing, keep an older replaced file beside it), then its settings and database (`09` §3 decision 26) | controller v0.269.0 (`backup/tier2_whole.go`) | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb + 5929b8e (D22, D25) | `audits/night-2026-09-24/A1/21-readback.json` (2026-09-24) |
|
||||
| <a id="b8"></a>[B8](capability-map-detail/B.md#b8) | A crash-looping or out-of-memory app is STOPPED by the box, the household and operator are told, and Start gives one more try (decision 28) | controller v0.269.0, hub v0.123.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A3/21-gokapi-page-event.txt` (2026-09-24) |
|
||||
| <a id="b9"></a>[B9](capability-map-detail/B.md#b9) | A held app whose page says support is informed can only be removed KEEPING its data (decision 27) | controller v0.269.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A2/10-held-no-copy-keep-data.json` (2026-09-24) |
|
||||
| <a id="b10"></a>[B10](capability-map-detail/B.md#b10) | The box runs the exact TESTED image of a floating tag, and an installed app keeps its image until a guarded Update moves it (`09` §6.4 part 6, box half) | controller v0.269.1 (`stacks/digest.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/B/21-floating-tag-0.269.1.json` (2026-09-24) |
|
||||
| <a id="b11"></a>[B11](capability-map-detail/B.md#b11) | Automatic updates at night — the update leg (`09` §6.4 part 7) | controller v0.271.0 | **PROVEN-LIVE** | scratch 9202, six simulated nights; demo boxes in Part D | `audits/DRILL-night-2026-09-25.md` (2026-09-25) |
|
||||
| <a id="b11"></a>[B11](capability-map-detail/B.md#b11) | Automatic updates at night — the update leg (`09` §6.4 part 7) | controller v0.271.0 | **PROVEN-LIVE** | scratch 9202, six simulated nights; demo boxes in Part D; **re-check (2026-10-24):** mechanism changed after its evidence — 63dcccf (security_fix steps, 0.341.0) | `audits/DRILL-night-2026-09-25.md` (2026-09-25) |
|
||||
| <a id="b12"></a>[B12](capability-map-detail/B.md#b12) | What `restart` and `update` do to a deployed app whose compose file the catalog already moved | controller v0.235.0 | **PROVEN-LIVE** | HISTORY: the behaviour up to v0.234.0; changed 2026-09-06 — current behaviour is B17 | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
|
||||
| <a id="b13"></a>[B13](capability-map-detail/B.md#b13) | Whether the box UPGRADES an app by itself, with nobody pressing anything | controller | **PROVEN-LIVE** | answer: yes, only when an app fails to come back (measured 2026-09-01) | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
|
||||
| <a id="b14"></a>[B14](capability-map-detail/B.md#b14) | Whether an app UPGRADE can be undone | controller + catalog | **PROVEN-LIVE** | answer: no, once a migration has run | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
|
||||
@@ -124,6 +124,7 @@
|
||||
| <a id="b20"></a>[B20](capability-map-detail/B.md#b20) | App crashes → customer notified (one event per transition, no flapping spam) | controller v0.120, hub v0.48 | **IMPLEMENTED** | crash→customer mail never live-confirmed | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
|
||||
| <a id="b21"></a>[B21](capability-map-detail/B.md#b21) | Post-deploy optional config (API keys etc.) with restart | controller, catalog `.felhom.yml` | **IMPLEMENTED** | config-save + restart in no campaign | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
||||
| <a id="b22"></a>[B22](capability-map-detail/B.md#b22) | Backup classification: 13 bind-bearing apps carry mandatory/optional/excluded classes | catalog, controller v0.132–133 | **PROVEN-LIVE** | — | `audits/SPIKE-backup-classification-2026-07-14.md` (2026-07-14); `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15) |
|
||||
| <a id="b23"></a>[B23](capability-map-detail/B.md#b23) | An urgent security fix reaches the boxes at night — a `security_fix` ladder step (box walk required, bench optional, a CVE/GHSA notice), taken by the night update (`09` §3 decision 255, D39 = B) | catalog `3898593`, controller v0.341.0 | **PROVEN-LIVE** | scratch 9202 (drill catalog), Plex 1.43.4 → 1.43.5; an app with no box fixture is checked by health, version and a soak only (`seed: none`) | `audits/security-lane-2026-10-23/` (2026-10-23) |
|
||||
|
||||
## C. Protection & recovery (the product promise)
|
||||
|
||||
@@ -146,10 +147,10 @@
|
||||
|---|---|---|---|---|---|
|
||||
| <a id="c1"></a>[C1](capability-map-detail/C.md#c1) | Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded DEGRADED rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | — | `audits/E2D-fresh-vm-2026-07-29.md` (2026-07-29) |
|
||||
| <a id="c2"></a>[C2](capability-map-detail/C.md#c2) | Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) |
|
||||
| <a id="c3"></a>[C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
|
||||
| <a id="c4"></a>[C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/VALIDATION-offbox-storagebox-2026-07-09.md` (2026-07-09) |
|
||||
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
|
||||
| <a id="c6"></a>[C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) |
|
||||
| <a id="c3"></a>[C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 03c0a69 + 66a69d4 + 5929b8e (capped container R-1214, kept copies D25) | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
|
||||
| <a id="c4"></a>[C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 8f5bea0 + d5f88a2 (abandon R-1182, off-site proof R-1189) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/VALIDATION-offbox-storagebox-2026-07-09.md` (2026-07-09) |
|
||||
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 93fb251 + d582feb (D20, D22) **2026-10-25:** changed again by controller 0.343.0 (D48 pull-first, R-1270 fix A: the verdict needs a stable start, the undo copy goes after 3 more minutes); on 0.343.0 Tester 1's local unit restore and set-aside restore passed and released the copy (`audits/release-2026-10-26/t1-restores-0.343.0/`); this path itself not re-walked. | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
|
||||
| <a id="c6"></a>[C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d5f88a2 + 5929b8e (R-1187 c, R-1191) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) |
|
||||
| <a id="c7"></a>[C7](capability-map-detail/C.md#c7) | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `pilot/DRILL-GL6-2026-07-08.md` (2026-07-08) |
|
||||
| <a id="c8"></a>[C8](capability-map-detail/C.md#c8) | PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | scoped to one host | `audits/SPIKE-pbsdr-selfheal-2026-07-15.md` (2026-07-15) |
|
||||
| <a id="c9"></a>[C9](capability-map-detail/C.md#c9) | Box survives an unattended app or guest-network failure (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0→v0.190.0, agent v0.92.1 | **PROVEN-LIVE** | demo box; boot-orphan leg 6 of 6 hard resets | `audits/DIAG-bootrecon-drive-absent-2026-08-02.md` (2026-08-02) |
|
||||
@@ -171,17 +172,17 @@
|
||||
| <a id="d3"></a>[D3](capability-map-detail/D.md#d3) | Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14); `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
|
||||
| <a id="d4"></a>[D4](capability-map-detail/D.md#d4) | NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | — | `audits/SPIKE-nas-verify-2026-07-11.md` (2026-07-11); `audits/SPIKE-nas-dsm-2026-07-11.md` (2026-07-11) |
|
||||
| <a id="d5"></a>[D5](capability-map-detail/D.md#d5) | Network storage (NAS) is browse + bulk-media only — it may NOT host an app's data namespace | controller v0.187.0 | **PROVEN-LIVE** | — | `audits/R108-network-app-namespace-2026-07-30.md` (2026-07-30) |
|
||||
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | — | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
|
||||
| <a id="d7"></a>[D7](capability-map-detail/D.md#d7) | A restore SAYS what it returned, and refuses what it cannot do — the four restore-surface truth defects from the 2026-08-21 drill | controller v0.226.0 (R-353, R-357, R-358, R-360, R-396) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: three of four live; R-357 implemented only | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (2026-08-30) |
|
||||
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb (D22, R-1178) **2026-10-25:** changed again by controller 0.343.0 (D48 pull-first, R-1270 fix A: the verdict needs a stable start, the undo copy goes after 3 more minutes); on 0.343.0 Tester 1's local unit restore and set-aside restore passed and released the copy (`audits/release-2026-10-26/t1-restores-0.343.0/`); this path itself not re-walked. | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
|
||||
| <a id="d7"></a>[D7](capability-map-detail/D.md#d7) | A restore SAYS what it returned, and refuses what it cannot do — the four restore-surface truth defects from the 2026-08-21 drill | controller v0.226.0 (R-353, R-357, R-358, R-360, R-396) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: three of four live; R-357 implemented only **2026-10-25:** controller 0.343.0 adds two truths here — a restore whose app does not start says so and keeps its copy (R-1270, D49 = A; live on 9202 rc.6), and a `.fab` import that cannot download its version refuses before it touches the app (R-1279, live). | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (2026-08-30) |
|
||||
| <a id="d8"></a>[D8](capability-map-detail/D.md#d8) | The box PROVES its own off-site copy still HOLDS something — a backup that is intact and EMPTY is caught without a person | controller v0.231.0 (R-87) + hub v0.110.0 | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r87-offsite-proof-2026-08-31/` (2026-08-31) |
|
||||
| <a id="d9"></a>[D9](capability-map-detail/D.md#d9) | The off-site store is VERIFIED on a cadence — something checks that the customer's backups are still readable | controller v0.228.0 (R-359, R-397, R-399) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r359-integrity-2026-08-30/` (2026-08-30) |
|
||||
| <a id="d10"></a>[D10](capability-map-detail/D.md#d10) | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | — | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13) |
|
||||
| <a id="d11"></a>[D11](capability-map-detail/D.md#d11) | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | — | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
|
||||
| <a id="d10"></a>[D10](capability-map-detail/D.md#d10) | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 (R-1124) | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13) |
|
||||
| <a id="d11"></a>[D11](capability-map-detail/D.md#d11) | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 + 5c3b218, controller 6c6bd87 (R-1124, R-1144) | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
|
||||
| <a id="d12"></a>[D12](capability-map-detail/D.md#d12) | Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13); `audits/CAMPAIGN-6A-2026-07-14.md` (2026-07-14) |
|
||||
| <a id="d13"></a>[D13](capability-map-detail/D.md#d13) | Media apps see EVERY drive; a drive added, pulled, back or safely removed is handled per app (notice, real restart, wiring, torrents paused/resumed) | controller v0.313.0, catalog `0188430`, hub v0.151.0 | **PROVEN-LIVE** | Tier-0 boxes (Tester 1, demo-hp 9201); the fleet with golden 0.313.0 + the floor | `audits/media-build-2026-10-15/G/` (2026-10-15); `audits/media-drive-events-2026-10-14/` (2026-10-14) |
|
||||
| <a id="d14"></a>[D14](capability-map-detail/D.md#d14) | A second local copy COUNTS only on a proven different physical disk: the agent knows each place's disks; a VM box makes the copy but says it cannot check; a same-disk copy is not made; a NAS never hides the one-copy notice; a box with no drive refuses a data-folder app in words | agent v0.157.0, controller v0.317.0, hub v0.154.0 | **PARTIAL** | live: VM `unknown` (Tester 1) and two real-disk boxes `proven` (demo-hp, N100); unit only: the same-disk skip, the NAS notice, the no-drive sentence; `usb_bridge` never seen on a device | `audits/storage-identity-buildA-2026-10-15/README.md` (2026-10-15); `architecture/16-storage-physical-identity.md` §13 |
|
||||
| <a id="d15"></a>[D15](capability-map-detail/D.md#d15) | A one-disk box gets its own data area („Belső tárhely") outside every whole-box copy, registered as the default drive with no press, so media and data-folder apps install with no extra step; a free disk's partitions can be formatted or used one by one | installer 1.33.0, agent v0.158.1, controller v0.318.0, golden 0.318.0 | **PROVEN-LIVE** | fresh 512 GB one-disk VM on demo-hp; partitions on a scratch disk; the old-box signed job and the D4 sentence unit-proven only; an empty part asks for confirmation (R-1093) | `audits/storage-identity-buildB-2026-10-15/README.md` (2026-10-16) |
|
||||
| <a id="d16"></a>[D16](capability-map-detail/D.md#d16) | A failed or cut restore puts the household's data back EXACTLY as it was and starts the app — local unit, second drive, kept load and off-site, and after a power cut or a killed controller (D22) | controller `main` (0.338.0-rc.1), unreleased | **IMPLEMENTED** | proven live on 9202 (rc.1): a damaged unit and a controller killed between the volume step and the replay; the second-drive pre-check (R-1178), the mirror skip (R-1180) and the off-site power cut by test only; a database outside the replaced volumes is not covered | `audits/night-2026-10-21/P3-restore/02-run.log` (2026-10-21) |
|
||||
| <a id="d16"></a>[D16](capability-map-detail/D.md#d16) | A failed or cut restore puts the household's data back EXACTLY as it was and starts the app — local unit, second drive, kept load and off-site, and after a power cut or a killed controller (D22) | controller 0.338.0 (D22) → **0.343.0** (released 2026-10-25: R-1251 a copy on an unbound drive waits; fix A) | **IMPLEMENTED** | proven live on 9202 (rc.1): a damaged unit and a controller killed between the volume step and the replay; the second-drive pre-check (R-1178), the mirror skip (R-1180) and the off-site power cut by test only; a database outside the replaced volumes is not covered **2026-10-25 (re-check):** released in 0.343.0; the release run proved the success path's copy release on Tester 1, not a failure put-back on the release build — the status stays. | `audits/night-2026-10-21/P3-restore/02-run.log` (2026-10-21) |
|
||||
|
||||
## E. Access, networking & household use
|
||||
|
||||
@@ -203,7 +204,7 @@
|
||||
| <a id="e14"></a>[E14](capability-map-detail/E.md#e14) | Multiple household users / per-person accounts | — | **MISSING** | — | R-15 |
|
||||
| <a id="e15"></a>[E15](capability-map-detail/E.md#e15) | A second login step for the dashboard (a TOTP code or a passkey) | — | **MISSING** | — | R-811 |
|
||||
| <a id="e16"></a>[E16](capability-map-detail/E.md#e16) | The household can leave Felhom, or outlive it — the box runs without the hub, the household owns its domain, tunnel and off-site account, and can export everything | — | **MISSING** | as a written answer | R-810 |
|
||||
| <a id="e17"></a>[E17](capability-map-detail/E.md#e17) | The host agent cannot reach root without the operator key: exact sudo patterns, no agent-written file installed where root reads it without a content check, the agent binary only by an operator-signed update checked as root | agent v0.146.1 (R-861; delivered by a step bundle, R-880) | **PROVEN-LIVE** | both demo boxes; three named residuals | `audits/hub-safety-2026-10-05/partF/` (2026-10-05) |
|
||||
| <a id="e17"></a>[E17](capability-map-detail/E.md#e17) | The host agent cannot reach root without the operator key: exact sudo patterns, no agent-written file installed where root reads it without a content check, the agent binary only by an operator-signed update checked as root | agent v0.146.1 (R-861; delivered by a step bundle, R-880) | **PROVEN-LIVE** | both demo boxes; three named residuals; **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 + a0c88e5 (sudoers + priv-apply, D4 drop-in) | `audits/hub-safety-2026-10-05/partF/` (2026-10-05) |
|
||||
| <a id="e18"></a>[E18](capability-map-detail/E.md#e18) | WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | — | `audits/SPIKE-oob-wg-operator-peer-2026-07-05.md` (2026-07-05); `audits/SPIKE-felhom-sshd-2026-07-05.md` (2026-07-05) |
|
||||
| <a id="e19"></a>[E19](capability-map-detail/E.md#e19) | The operator can see WHERE a managed host is — its LAN address and its WireGuard address, on the host page | agent v0.119.0, hub v0.85.0 | **PROVEN-LIVE** | — | `audits/host-addresses-visible-2026-07-31.md` (2026-07-31) |
|
||||
| <a id="e20"></a>[E20](capability-map-detail/E.md#e20) | The operator can see whether a managed host's guests still have working networking — and how often the watchdog had to repair them | agent v0.92.0 (emitter, 2026-07-21), hub v0.104.0 (reader, 2026-08-13) | **IMPLEMENTED** | not proven live | R-319 |
|
||||
@@ -222,6 +223,7 @@
|
||||
| <a id="f7"></a>[F7](capability-map-detail/F.md#f7) | Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
|
||||
| <a id="f8"></a>[F8](capability-map-detail/F.md#f8) | Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | no corpus validation doc | operator infra; no evidence document |
|
||||
| <a id="f9"></a>[F9](capability-map-detail/F.md#f9) | Backup-deadline alerting (`expected_backup_missed`) is ANCHORED — absence of signal is UNKNOWN, not failure | hub v0.75.0 | **IMPLEMENTED** | — | `audits/DIAG-backup-missed-2026-07-26.md` (2026-07-26); R-82 |
|
||||
| <a id="f13"></a>[F13](capability-map-detail/F.md#f13) | A hacked app is noticed: an app that uploads or computes far more than its own numbers explain alarms the operator only; each box tests its upload line weekly and qBittorrent gets half of it (`08` §9; D41, D42; R-1244, R-1247) | controller v0.342.0, agent v0.164.0, hub v0.169.0, ep0 `felhom-iperf3` | **PROVEN-LIVE** | the alarm path on scratch 9202 (a hidden flood and a CPU burner, 2026-10-23); the line test on the three test boxes; the qBittorrent limit on demo-felhom. **Gaps:** a seeding qBittorrent is never tested (R-1250); an app's API can lie (R-1248) | `audits/SPIKE-upload-watch-2026-10-23.md` §7, `audits/line-test-2026-10-24/` (2026-10-24) |
|
||||
| <a id="f12"></a>[F12](capability-map-detail/F.md#f12) | After a power cut, an app rebuilds the files the cut left empty, and the household sees any lost photo on the app page | controller v0.311.0, hub v0.150.0, catalog Immich block | **PROVEN-LIVE** | demo-hp 2026-10-14: a real cut mid-upload, 41 of 41 broken photos repaired, 120 controls untouched; the lost-photo note not exercised live (0 lost; unit- and 9202-tested). **Fleet delivery done 2026-10-14:** golden 0.311.0 + global floor; all three demo boxes on 0.311.0; catalog gate `unclean-boot-images` | `audits/immich-repair-2026-10-14/README.md` (2026-10-14); `09` §3 decision 236; R-1007 |
|
||||
| <a id="f11"></a>[F11](capability-map-detail/F.md#f11) | The household turns the whole machine off („A gép kikapcsolása"); the hub holds every silence alarm for 14 days and shows KIKAPCSOLVA | agent v0.155.0, hub v0.149.0, controller v0.310.0 | **PROVEN-LIVE** | proven on the N100 2026-10-13; fleet delivery waits on the floor | `audits/poweroff-2026-10-13/README.md` (2026-10-13); `architecture/14-host-power-off.md`; R-1000, R-1002 |
|
||||
| <a id="f10"></a>[F10](capability-map-detail/F.md#f10) | A box that is not always on (off at its backup time) makes the missed night up once when it comes back; the household sees a banner; the operator gets the missed-backup alarm, down or not | controller v0.295.0, hub v0.134.0, agent v0.145.0 | **PARTIAL** | catch-up and banner live; R-872 alarm by tests; suspend not measured | `audits/catchup-2026-10-05/partA/` (2026-10-05); R-872 |
|
||||
@@ -236,7 +238,7 @@
|
||||
| <a id="g4"></a>[G4](capability-map-detail/G.md#g4) | Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules | hub v0.45/0.53, agent | **PARTIAL** | — | `runbooks/publish-train-rules.md` (2026-09-13); R-23 |
|
||||
| <a id="g5"></a>[G5](capability-map-detail/G.md#g5) | Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed | agent v0.70+ | **PROVEN-LIVE** | demo | `audits/SPIKE-agent-selfupdate-2026-07-05.md` (2026-07-05) |
|
||||
| <a id="g6"></a>[G6](capability-map-detail/G.md#g6) | Controller self-update: anonymous registry, no credentials in guest | controller v0.112 | **PROVEN-LIVE** | demo; the file found is the floor-driven self-update leg | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
||||
| <a id="g7"></a>[G7](capability-map-detail/G.md#g7) | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | — | `audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md` (2026-07-09); `audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (2026-07-09) |
|
||||
| <a id="g7"></a>[G7](capability-map-detail/G.md#g7) | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — hub 9688e76e (scoped abandonment, R-1182) | `audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md` (2026-07-09); `audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (2026-07-09) |
|
||||
| <a id="g8"></a>[G8](capability-map-detail/G.md#g8) | Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **IMPLEMENTED** | no live-fired leg | `audits/CAMPAIGN-offsite-overnight-2026-07-10.md` (2026-07-10) |
|
||||
| <a id="g9"></a>[G9](capability-map-detail/G.md#g9) | Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert) | hub v0.64.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.64.0 code and unit tests (outside `documentation/`) |
|
||||
| <a id="g10"></a>[G10](capability-map-detail/G.md#g10) | Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge) | hub v0.65.0 + tenantsync v1.2.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.65.0 code and unit tests (outside `documentation/`) |
|
||||
@@ -244,10 +246,10 @@
|
||||
| <a id="g12"></a>[G12](capability-map-detail/G.md#g12) | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | — | the 2026-07-13 hub closing bundle (no evidence document) |
|
||||
| <a id="g13"></a>[G13](capability-map-detail/G.md#g13) | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | — | hub v0.54 (2026-07-13; no evidence document) |
|
||||
| <a id="g14"></a>[G14](capability-map-detail/G.md#g14) | The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail | hub v0.135.0 (R-604, R-530) | **IMPLEMENTED** | the page live; the alarm and the mail unit-proven | `audits/hub-safety-2026-10-05/partD/` (2026-10-05) |
|
||||
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | — | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a consistent copy every 6 hours since D34 (nightly before 2026-10-23; corrected 2026-10-24), encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||
| <a id="g16"></a>[G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||
| <a id="g17"></a>[G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) |
|
||||
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
|
||||
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
|
||||
| <a id="g19"></a>[G19](capability-map-detail/G.md#g19) | An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps | controller v0.259.0 + hub v0.119.0 + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE** | walked on 0.258.0 with one blocker; fixed, the walk not repeated | `audits/DRILL-first-hour-en-0258-2026-09-20.md` (2026-09-20); `audits/i18n-closing-2026-09-21/live/` (2026-09-21) |
|
||||
| <a id="g20"></a>[G20](capability-map-detail/G.md#g20) | A deletion of a customer's off-site history is NOTICED within a day | hub v0.111.0 (R-431) | **IMPLEMENTED** | not yet proven live | R-431 |
|
||||
|
||||
|
||||
@@ -16,15 +16,17 @@
|
||||
|
||||
|
||||
**Status:** draft (decisions from the topology/trust design sessions).
|
||||
**Platform facts** referenced here live in `docs/proxmox-platform.md`; this document
|
||||
**Platform facts** referenced here live in `documentation/proxmox-platform.md` (path corrected 2026-10-24); this document
|
||||
records *Felhom's decisions*, not Proxmox behaviour.
|
||||
|
||||
---
|
||||
|
||||
## 1. Model at a glance
|
||||
|
||||
Three components. **Control is always box-initiated** — the hub never connects *into* a
|
||||
customer box.
|
||||
Three components. **Control is always box-initiated** — the hub never ~~connects~~ sends a command *into* a
|
||||
customer box. **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The hub may POKE a box: through ep0, one empty UDP datagram to the box's WireGuard
|
||||
address (port 51822; `hub/internal/poke/client.go`, agent `internal/poke/poke.go`). A poke carries no command; it only
|
||||
makes the box pull its desired state at once. The box still pulls.
|
||||
|
||||
```
|
||||
operator side customer box (per Proxmox host)
|
||||
@@ -98,14 +100,18 @@ credentials.
|
||||
|
||||
- CGNAT does **not** force this: the Cloudflare Tunnel already makes a box reachable through
|
||||
Cloudflare's edge. We *choose* box-initiated control for the smallest attack surface — the
|
||||
box exposes no control endpoint at all.
|
||||
- The agent and the controller **poll** the hub; the hub never initiates inbound.
|
||||
box exposes no control endpoint ~~at all~~: no route on the box takes a command from the hub. **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]**
|
||||
Two listeners sit behind the WireGuard tunnel: the agent's poke listener (a poke carries no command; the box still
|
||||
pulls) and the operator door, `felhom-sshd` on the host, reachable only from the operator's `/32` peer — the
|
||||
documented break-glass (`03` §5, `06` §4.5).
|
||||
- The agent and the controller **poll** the hub; the hub never initiates ~~inbound~~ a command inbound — a poke only
|
||||
makes them poll at once (corrected 2026-10-25, D45, R-1266).
|
||||
- Operator actions are delivered as **signed jobs**: the agent verifies an operator signature
|
||||
before executing, so a compromised hub database alone cannot forge commands.
|
||||
- **[FACT, 2026-10-16, R-970]** The last contradiction is gone: until controller v0.322.0 the box's own hub API key,
|
||||
sent as a Bearer token, opened `/api/config/` (read `controller.yaml` with every secret in it, and REPLACE it,
|
||||
unsigned) and the self-update and geo routes. Nothing called them (the hub never connects to a box). The config
|
||||
routes are removed and no route accepts the hub key; every `/api/` route needs a dashboard session (pinned by
|
||||
routes are removed and no route accepts the hub key; every `/api/` route but the read-only `/api/health` needs a dashboard session (pinned by
|
||||
`TestR970_HubKeyOpensNoDoor` on the real route table). The box exposes no control endpoint, as stated above.
|
||||
- All operator-initiated actions are recorded in a **customer-visible audit log**.
|
||||
|
||||
@@ -132,7 +138,7 @@ of all 53 apps, 2026-09-28).
|
||||
**[DESIGN, 2026-10-17, R-1126, operator decision D11 = A] The tunnel runs before the claim, and FileBrowser never
|
||||
answers on a default password.** The box starts cloudflared as soon as it has a tunnel token, before it is claimed,
|
||||
because the claim itself is reached through it: the claim mail links `https://felhom.<domain>`
|
||||
(`hub/internal/notify/templates.go:27`). Before the claim a stranger reaches only the claim page (a code: 72 h life, 5
|
||||
(`hub/internal/notify/templates.go` `FormatClaimEmail`; line corrected 2026-10-24). Before the claim a stranger reaches only the claim page (a code: 72 h life, 5
|
||||
tries, 15-minute lockout — controller `web/claim.go`) and `files.<domain>`. FileBrowser starts every new database on
|
||||
admin/admin, and its own admin-password setting cannot replace that (it re-applies at every start and would overwrite
|
||||
the household's own password — FileBrowser 1.5.6 `backend/cmd/user.go:50-57`, measured). So FileBrowser starts OFF
|
||||
@@ -219,7 +225,10 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
||||
This removes any zero-touch remote-enrollment problem.
|
||||
- A **one-time retrieval code** mints durable identity. Single-use (burned on the successful
|
||||
config fetch) plus a short *pre-use* TTL; one-click regenerate for the only real failure
|
||||
case (fetch fails before anything is persisted). After the fetch, the code is irrelevant —
|
||||
case (fetch fails before anything is persisted). *(As built — corrected 2026-10-24: the passphrase is not burned
|
||||
and has no TTL; it gives keys until the box has a host record AND a controller ≥ 0.334.0 has reported with its own
|
||||
key, then `/config`, `/recovery` and `/host-enroll` answer 409 — D19, R-1159, `hub/internal/api/passphrase_scope.go:19-30`;
|
||||
`05` §8 has the full rule.)* After the fetch, the code is irrelevant —
|
||||
everything downstream runs on durable credentials, so retries don't need it.
|
||||
- **Order:** the agent enrolls first (and, running as root at setup, mints its own scoped
|
||||
operator-tier Proxmox token), then provisions the customer LXC from the golden template and
|
||||
@@ -242,7 +251,8 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
||||
2026-10-08, R-900). Replacing it with our own relay is deferred until after the first customers (R-904, `09` §3
|
||||
decision 184).
|
||||
- **Outbound only** for control/report/backup (poll to hub, push to PBS). No inbound control
|
||||
endpoint exists in the chosen model.
|
||||
endpoint exists in the chosen model. **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** A poke carries no command; the box still pulls. The operator
|
||||
door (`felhom-sshd` over the WireGuard tunnel) is the documented break-glass (§4, `03` §5).
|
||||
- **Every customer has their OWN domain — never a name under `felhom.eu`** (operator ruling
|
||||
2026-09-14). The free Cloudflare tier's certificate covers one level below a zone, so nested names
|
||||
such as `felhom.<customer>.felhom.eu` are not covered; the customer's own domain is a few thousand
|
||||
@@ -253,7 +263,7 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
||||
2026-09-14 first-hour drill used a `*.felhom.eu` customer domain with no tunnel, and the dashboard link
|
||||
in the setup-code mail did not resolve.
|
||||
- **A pasted Cloudflare API token must reach exactly the customer's own zone** (R-138 option C, `09` §3 decision 190;
|
||||
hub main 2026-10-08, unreleased). On save the hub asks Cloudflare `GET /zones` with the token and keeps it only when
|
||||
released in hub v0.144.0 — corrected 2026-10-24). On save the hub asks Cloudflare `GET /zones` with the token and keeps it only when
|
||||
the token sees exactly ONE zone and that zone IS the customer's domain (a zone above it is refused: a sibling
|
||||
customer added later would be reachable). Cloudflare unreachable → the save fails and the old token stays. The token
|
||||
is never logged. **Limit:** `/zones` lists what the token can READ; a hand-built token with Zone:Read on one zone and
|
||||
@@ -313,7 +323,8 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
||||
>
|
||||
> **What being unmarked cost.** The consequence of this bullet — that 40 of 53 catalogue templates
|
||||
> declare no configurable path because they are all-hot — is stated as **[FACT]** at
|
||||
> `07-backup-architecture.md:296-299`. A reader met a marked observation beside an unmarked choice
|
||||
> `07-backup-architecture.md:296-299`. *(Dated note 2026-10-24: that pointer is dead — those lines are about the local
|
||||
> plaintext posture now, and no architecture file but this one states „40 of 53 … all-hot".)* A reader met a marked observation beside an unmarked choice
|
||||
> and reasonably asked whether it *should* be so. **Between 19 and 22 August that reader called this
|
||||
> decision a defect in four places** (R-370), and one session's work went into correcting the record
|
||||
> rather than into the product.
|
||||
@@ -326,7 +337,12 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
||||
- **Key custody (zero-knowledge default):** three tiers the customer chooses —
|
||||
*customer-only* / *zero-knowledge escrow (default)* / *operator-managed*. Default escrows
|
||||
the **PBS passphrase-protected keyfile** in the hub, wrapped under a **customer recovery
|
||||
code** the operator can't open; DR needs the customer's code. Access-notification is an
|
||||
code**; DR needs the customer's code. ~~the operator can't open~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The hub keeps only the
|
||||
locked key package, and every older one it retained; without the customer's code the operator cannot open any of
|
||||
them, and from the off-site storage and the hub nobody can read the backup. Two exact limits, as the website says
|
||||
since R-985: while the box runs, the operator has root access for support and with it could reach the data and the
|
||||
backup key on the box; and the operator can open an OLDER off-site store from a retained package, by hand, when the
|
||||
household types its old code on its own box (`runbooks/RUNBOOK-open-retained-package.md`, R-304). Access-notification is an
|
||||
audit signal, never the primary guard. (Don't build bespoke crypto — use PBS's native
|
||||
keyfile passphrase.)
|
||||
|
||||
@@ -343,14 +359,19 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
||||
restores from PBS; the controller returns on the same domain (tunnel reused from the hub
|
||||
record). DR = provisioning + a restore mode, not a separate mechanism.
|
||||
- **Snapshot-before-deploy:** controller asks the agent to snapshot, deploys, runs its
|
||||
post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.)
|
||||
post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.) **[DESIGN — not built,
|
||||
noted 2026-10-24]:** the agent serves `/snapshot` and `/rollback`, but no controller code calls them
|
||||
(`felhom-controller/controller/internal/agentapi`); the update's guard is the per-app backup + undo of `09` §3 decision 15.
|
||||
|
||||
---
|
||||
|
||||
## 10. How this embodies the product values
|
||||
|
||||
- **Zero-knowledge offsite** — the operator holds the offsite backup but cannot read it.
|
||||
- **Box-initiated control + signed jobs** — no standing operator backdoor; a hub compromise
|
||||
- **Zero-knowledge offsite** — the operator holds the offsite backup but cannot read it without the customer's code
|
||||
(the two exact limits are in §8 — corrected 2026-10-25, D45, R-1266).
|
||||
- **Box-initiated control + signed jobs** — ~~no standing operator backdoor~~ no hidden operator door: the one
|
||||
operator door is `felhom-sshd` over the WireGuard tunnel, the documented break-glass (`03` §5), and the website tells
|
||||
the household about the operator's root access (R-985) — corrected 2026-10-25, D45, R-1266; a hub compromise
|
||||
alone can't forge commands.
|
||||
- **Customer-visible audit log** — every operator action is visible to the customer.
|
||||
- **Never hold data hostage** — subscriptions cover ongoing labour (monitoring, offsite,
|
||||
|
||||
@@ -105,7 +105,9 @@ Tri-state: `""` (unknown) · `"running"` · `"stopped"`.
|
||||
|
||||
### In-flight — a marker file, one per owner
|
||||
|
||||
Two exist and they are deliberately **separate files**: `quiesce-state.json` (the whole-guest backup
|
||||
Two exist and they are deliberately **separate files** *(four in-flight records by 2026-10-24: also `update-journal.json`
|
||||
— the guarded update's crash journal read by `RecoverUpdates`, `internal/stacks/update.go:49,1306` — and
|
||||
`.felhom-install-pending` — an interrupted install, R-681, `install_interrupted.go:28`; noted 2026-10-24)*: `quiesce-state.json` (the whole-guest backup
|
||||
window, `internal/quiesce`) and `appstop-state.json` (app-data operations that stop an app —
|
||||
`backup.AppStopGuard`, covering the volume dump, offbox reconstitution and `.fab` export). **One
|
||||
file, one writer**: sharing would give one record two lifetimes, and one owner clearing the other's
|
||||
@@ -169,7 +171,8 @@ A recovery completing after the grace emits a `LATE RECOVERY` warning naming the
|
||||
|
||||
**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
|
||||
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
|
||||
answer it, and the last two only became reachable once the window widened past T+5 s:
|
||||
answer it *(four since slice 4: check „1b", a guarded update moving the app — `IsUpdating`, `cmd/controller/main.go:2459-2477`;
|
||||
noted 2026-10-24)*, and the last two only became reachable once the window widened past T+5 s:
|
||||
|
||||
| Holder | Why starting would be wrong |
|
||||
|---|---|
|
||||
@@ -218,6 +221,13 @@ own; every caller that is not the customer must decide for itself whether the ap
|
||||
|
||||
---
|
||||
|
||||
> *Dated note 2026-10-24 (review round 2): this inventory is v0.33's. On `main` (`321df1f`) 22 packages under
|
||||
> `controller/internal/` have no row here: `aftercrash`, `agentapi`, `appbackup`, `backupwindow`, `bootrecon`,
|
||||
> `bootstrap`, `channelhealth`, `crashboot`, `dockerexec`, `family`, `fillwatch`, `i18n`, `infra`, `logx`, `mailrelay`,
|
||||
> `mediawire`, `nightchain`, `offsiteapply`, `privpolicy`, `quiesce`, `sockheal`, `upwatch` (an `ls` of the tree). Each
|
||||
> package's own doc comment is its description; `sockheal` (re-heal after the guest's Docker socket is recreated,
|
||||
> R-860) and `channelhealth` (a periodic proof of the controller→agent channel) are in no architecture document.*
|
||||
|
||||
## 2. Classification table (per package/file)
|
||||
|
||||
### `cmd/`
|
||||
@@ -266,6 +276,11 @@ own; every caller that is not the customer must decide for itself whether the ap
|
||||
|---|---|---|---|
|
||||
| `client.go`,`zone.go`,`waf.go`,`geosync.go`,`countries.go` | **DELETE (→hub)** | The **hub** holds the CF API token and reconciles geo desired-state → WAF (doc 01 §5, doc 03 §2). The controller no longer calls the Cloudflare API — it reports geo desired-state up. The customer-facing geo *preference UI/data* stays (see `api/geo.go`). | needs-rework |
|
||||
|
||||
*As built (noted 2026-10-24, review round 2): the `cloudflare/` package is still wired — with `cf_api_token` set the
|
||||
controller builds a `GeoSyncManager` and calls the Cloudflare API (`cmd/controller/main.go:1992-2000`; the field is still in
|
||||
`config.go:160`). The DELETE classification above is the plan, not the state; whether any box sets the token is not
|
||||
readable from source.*
|
||||
|
||||
### `config/`, `crypto/`, `util/`
|
||||
| File | Class | Reason | Risk |
|
||||
|---|---|---|---|
|
||||
@@ -325,6 +340,7 @@ household's file names. The household's lost list is shown on the app page (`los
|
||||
| File | Class | Reason | Risk |
|
||||
|---|---|---|---|
|
||||
| `version.go` | **KEEP** | Semver parse / version string (still used for reporting). | clean |
|
||||
| *(as built 2026-10-24: both files below are LIVE — the updater decides the version and the floor and hands the container swap to the agent, `AgentSwapper`, `internal/selfupdate/updater.go:38-47`; `state.go` keeps `update-state.json`)* | | | |
|
||||
| `state.go` | **DELETE (obsolete)** | Self-update audit state — the agent owns controller updates now (doc 03 §11). | clean |
|
||||
| `updater.go` | **DELETE (→agent)** | Resolved (doc 03 §11): the controller is **agent-managed** — the agent snapshots → redeploys → health-gates → rolls back the controller. The controller's old self-update path (image pull + compose edit) is **removed**. | clean |
|
||||
|
||||
@@ -387,7 +403,8 @@ dashboard alert is an `Alert` with two placement fields: `PageOnly` (the pages i
|
||||
page) and `Inline` (rendered by the page template in place, not by the layout's banner). `GetAlerts` returns
|
||||
the list (endpoint-drift, agent-channel and dead-app alerts first, then the rest sorted error > warning > info,
|
||||
capped at five plus an overflow line), and `layout.html` paints only those that are not `Inline` and match the
|
||||
page; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones.
|
||||
page *(as built since R-906 — noted 2026-10-24: pages use `GetBannerAlerts(page, lang)`, which filters per page BEFORE the
|
||||
cap, `internal/web/alerts.go:353`, `handlers.go:144`)*; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones.
|
||||
|
||||
**Exactly one warning is inline today:** the „storage is not on a separate drive" health warning. It is
|
||||
`PageOnly: dashboard, monitoring` and `Inline: true`, so it sits quietly under the storage bars; **every other
|
||||
@@ -821,7 +838,7 @@ R-438. The three facts below are the ones a reader needs before touching any of
|
||||
`Syncer.copyTemplates` (`sync/sync.go:319`) walks every directory in the catalog cache and copies
|
||||
`docker-compose.yml` and `.felhom.yml` into the matching stack folder. **There is no test of whether
|
||||
the app is deployed.** The only guard is a sha256 content compare (`copyIfChanged`) and the only
|
||||
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351`), plus
|
||||
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351` then, `:399` now; `copyTemplates` is now `sync.go:436`), plus
|
||||
one immediate sync at controller start (`sync.go:98`).
|
||||
|
||||
**SINCE v0.235.0** this walk still happens and `.felhom.yml` is still copied unconditionally, but the
|
||||
@@ -835,6 +852,7 @@ and they stay that way until something else acts.
|
||||
|
||||
### 2. Every lifecycle path resolves that disagreement, silently, by upgrading
|
||||
|
||||
*(This section is the 2026-09 measurement; `UpdateStack` was deleted in v0.237.0 — the button calls `StartGuardedUpdate`, `09` §6.1 — noted 2026-10-24.)*
|
||||
`StartStack`, `RestartStack` and `UpdateStack` (`stacks/manager.go:1029 / 1133 / 1170`) all end in
|
||||
`docker compose up -d`. `up -d` makes the container match the file, **and pulls the image itself if it
|
||||
is absent** — measured at 18.3 s with a pull versus 0.5 s without, against a negative control
|
||||
@@ -887,7 +905,9 @@ measured; that the restore writes to that path is read, not measured.
|
||||
beállítások csak olvashatók.", `deploy.html`). This is a design, not an oversight: a changed value would
|
||||
need a guarded re-deploy that nothing performs. **Finding (2026-09-15):** the catalog field flag
|
||||
`locked_after_deploy` (`stacks/metadata.go`) is parsed and read by **no** controller code — every field
|
||||
is read-only after install whatever the catalog says. So a page must never tell a household to change
|
||||
is read-only after install whatever the catalog says. *(Dated note 2026-10-24: it is now read into the app's
|
||||
`LockedFields` (`internal/stacks/deploy.go:106,534`) and enforced only by `UpdateStackConfig`, which nothing calls — so
|
||||
„read-only whatever the catalog says" still holds in practice.)* So a page must never tell a household to change
|
||||
a value after install. **Vaultwarden (R-512)** follows the design instead of breaking it: registration
|
||||
is closed by default and the household is invited from the admin panel, measured to work without mail
|
||||
(stranger 400, invite 200, invited 200).
|
||||
|
||||
@@ -47,7 +47,7 @@ Owns:
|
||||
1. **Proxmox lifecycle** — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (the **`FelhomAgent` operator role** — `proxmox-platform.md` §3.6, validated Phase 3 B3) for everything the API covers; raw host ops only where unavoidable.
|
||||
2. **Storage management** — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests.
|
||||
3. **Backup/restore orchestration** — vzdump to the tiers, PBS, snapshot management, and the **self-restore-test**.
|
||||
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Three states: `running` (healthy), `not_running` (stopped, absent, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
|
||||
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Four states (corrected 2026-10-24): `running` (healthy), `not_running` (stopped, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms — and since agent v0.161.0 `absent` (the guest has no cloudflared container at all, R-1070; the hub never alarms on it) — `internal/hub/cloudflared.go:12-23`. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
|
||||
Proxmox API, plus the OS wrapper's read-only `facts` (host Debian, next-boot kernel, held packages, taint, the crash
|
||||
guard; guest Debian, Docker engine, containerd, live-restore), at most every 10 min. **Still no new sudoers line** — the
|
||||
facts, `live-restore-on` and the Docker layer all ride `FELHOM_OSAPPLY`. New ROOT-owned files instead (installer 1.30.0;
|
||||
@@ -64,7 +64,8 @@ Explicitly does **not**:
|
||||
- Serve application traffic or sit in the data path. **Control plane, not data plane**: if the agent dies, apps keep serving (Docker + LXC run without it); only *management* degrades — no new backups, no provisioning, hub loses the heartbeat.
|
||||
- Hold or proxy customer application data.
|
||||
- Run inside a guest. It is the thing that recovers guests and the host; it cannot be one of them.
|
||||
- Manage **geo-restriction / the Cloudflare API**. Geo is hub-owned: the customer sets it in the controller UI, the controller reports the geo desired-state to the hub, and the **hub** (holding the CF API token) reconciles the WAF (S4). The agent manages only the *tunnel* service (`cloudflared`, §3/§5), never WAF rules.
|
||||
- Manage **geo-restriction / the Cloudflare API**. Geo is hub-owned: the customer sets it in the controller UI, the controller reports the geo desired-state to the hub, and the **hub** (holding the CF API token) reconciles the WAF (S4). ~~The agent manages only the *tunnel* service (`cloudflared`, §3/§5), never WAF rules.~~ The agent manages no
|
||||
Cloudflare piece: it only reads the guest's `cloudflared` container (§3) — corrected 2026-10-25, D45, R-1266.
|
||||
|
||||
## 3. Process model & host integration
|
||||
|
||||
@@ -129,13 +130,21 @@ the one trust root; the agent's API token is untouched; a box gets the new rule
|
||||
`agent_config_update` (the order: signed `agent_update` first, then the bundle — after the bundle, an agent below
|
||||
0.146.0 cannot update itself on that box).
|
||||
|
||||
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — built as a ready patch, NOT shipped] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
|
||||
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — SHIPPED in agent v0.163.0 (`1109092`) and installer-v1.36.0 (`manifests/webpage.yaml:347`); corrected 2026-10-24] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
|
||||
**[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased; R-1257]** **Appliances only.** The
|
||||
drop-in's bundle policy is `appliance`: written only when the root-owned install record (`/var/lib/felhom-install/state.json`)
|
||||
says `appliance` (the installer records the mode before it installs the bundle); on a `byo` record nothing is written and
|
||||
a copy WE wrote (this path, beginning with our header line) is removed, then `sshd -t` and a reload; a box with no
|
||||
readable record is left as it is. The „in force" check asks `sshd -T` for each of the host's own global IPv4 addresses
|
||||
(a `Match Address <LAN>` block no longer reads as in force); a failing `sshd -t` puts back ONLY the SSH file and keeps
|
||||
the rest of the bundle (the report says `sshd_step_undone`).
|
||||
|
||||
**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
|
||||
**[FACT, agent `a0c88e5`, released in v0.163.0 — corrected 2026-10-24] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
|
||||
|
||||
**[DESIGN, operator ruling D30 2026-10-22; agent main `ae69ad4`, unreleased] The tunnel heals itself by a new key.** When
|
||||
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the old
|
||||
private key kept as `wg/private.key.prev`, 0600 — the escrow may hold it) and registers the new public key on the
|
||||
**[DESIGN, operator ruling D30 2026-10-22; agent `ae69ad4`, released in v0.163.0 — corrected 2026-10-24] The tunnel heals itself by a new key.** When
|
||||
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the FIRST old
|
||||
private key kept as `wg/private.key.prev`, later ones as `wg/private.key.prev.<unix>`, the newest three kept, 0600 — the
|
||||
escrow may hold one; the once-an-hour bound survives a restart in `wg/last-rekey` — `internal/wgtunnel/rekey.go`, corrected 2026-10-24) and registers the new public key on the
|
||||
existing path, `POST /hosts/<id>/wg`: the hub re-keys the peer in place (same /32, generation bump) and its full-list
|
||||
push to ep0 (`wgsync`, `wg syncconf`) drops the old key and adds the new one. A new key is a new peer for ep0, with no
|
||||
stored timestamp. Once an hour at most; not while the unit is down; never during a re-register already in flight. The
|
||||
@@ -232,10 +241,16 @@ alerts the operator when an agent misses its expected check-in window. This is t
|
||||
failure mode for a managed service, so it gets first-class treatment hub-side.
|
||||
|
||||
**Break-glass.** Standing inbound control is off. But when the poll loop *itself* is wedged
|
||||
(agent hung, host sick) you cannot fix it through the poll loop. So there is an explicit,
|
||||
**off-by-default, customer-consented, fully-audited** emergency path: SSH to the host via
|
||||
(agent hung, host sick) you cannot fix it through the poll loop. So there is an explicit emergency path.
|
||||
~~**off-by-default, customer-consented, fully-audited** emergency path: SSH to the host via
|
||||
the Cloudflare Tunnel behind Cloudflare Access (or on-site). Enabling it is itself a signed,
|
||||
logged operation; it auto-expires.
|
||||
logged operation; it auto-expires.~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The path is the operator door: the operator's WireGuard
|
||||
peer reaches a dedicated `felhom-sshd` on the host (a non-22 port, reachable only from the operator's `/32` over
|
||||
`wg-felhom`, key-only login; `06` §4.5, TASK H1). It does not use the Cloudflare Tunnel. It is installed by default on
|
||||
an appliance install, and a BYO install refuses it (the owner's call; `scripts/felhom-host-install.sh`). It stays on;
|
||||
it is not opened per incident. When SSH itself is broken, the hub-vaulted `root@pam` password opens the Proxmox web
|
||||
console (`runbooks/break-glass.md`). The website tells the household that the operator has root access for support
|
||||
(R-985).
|
||||
|
||||
## 6. Agent ↔ controller local API
|
||||
|
||||
@@ -250,8 +265,9 @@ The controller (in its LXC) reaches the agent (on the host) over the local bridg
|
||||
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
|
||||
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
|
||||
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy`
|
||||
(R-921, agent Unreleased `56aabc7`):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
|
||||
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off), the
|
||||
(R-921, agent `56aabc7`, released in v0.161.0 — corrected 2026-10-24):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
|
||||
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off, and since v0.164.0 the
|
||||
weekly upload line test — gate name `line-test`), the
|
||||
state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it
|
||||
before it stops any app. A read, never a reservation.
|
||||
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
|
||||
@@ -306,6 +322,49 @@ A controller can only `POST /rollback` (or snapshot/backup) **its own** guest
|
||||
token → guest and authorizes per guest, so a compromised controller's blast radius is
|
||||
**self-scoped and bounded** to its own guest.
|
||||
|
||||
**[DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265); every route listed below since
|
||||
2026-10-25]:** the surface listed above is the slice-8 design; on `main` the local API registers **41** routes
|
||||
(`felhom-agent/internal/localapi/server.go:562-634`). The list above names 15 of them; the other 26 are in the table
|
||||
below (recounted 2026-10-25 — the review counted 24). Every route is behind the same per-guest token (`withGuest`).
|
||||
Among them the controller can start a HOST action: `POST /host/line-test/run` (about 55 s; bounded by the due
|
||||
date, the window, 6 tries a day and the heavy-operation gate — R-1250, unreleased). The local API also listens on the
|
||||
R-50 island bridge (host `169.254.253.1`, guest `net1` `169.254.253.2/30`, `internal/config/config.go:260-269`). The
|
||||
weekly line test keeps its state in the OOB state dir (`line-test.json`) and has an off switch, `line_test.disable`.
|
||||
The served WireGuard block carries `oob_peer_ip` and `oob_operator_ssh_key` (`internal/hub/report.go`). The D30 re-key
|
||||
runs only inside the tunnel watchdog, and its 15-minute count lives in memory — an agent restart starts it again (the
|
||||
once-an-hour bound survives in `wg/last-rekey`).
|
||||
|
||||
The routes the list above does not name (read from `server.go:562-634` and each handler's own comment, 2026-10-25):
|
||||
|
||||
| Route | What it does |
|
||||
|---|---|
|
||||
| `GET /backup/tiers` | this guest's backup tiers, each with its cadence and whether its Proxmox storage exists now (`present` / `absent` / `unknown`) |
|
||||
| `POST /backup/target` | moves the primary backup tier to a new storage: create → grant → config; answers `restart_required`, and the caller restarts the agent after its own in-flight check |
|
||||
| `GET /host/line-test` | HOST: the weekly upload line test's last measurement; `present:false` when never measured (R-1247) |
|
||||
| `POST /host/line-test/run` | HOST: starts a line test now — about 55 s, bounded as above (R-1250) |
|
||||
| `GET /disks/candidates` | the host disks free for Felhom to enroll (`initialize`, and `attach` for those with a mountable file system) |
|
||||
| `POST /disks/same-disk` | whether two of the guest's paths sit on the same physical disk, from the agent's own read (`16` §4) |
|
||||
| `POST /disks/decommission` | takes the household's own drive out for good: a permanent intent, the guest bind pruned, unmounted; never formats; a system or backup mount is refused (403) |
|
||||
| `GET /disks/format/status` | the newest or running format job, for a caller whose request timed out |
|
||||
| `POST /disks/guest-attach` | binds an enrolled drive's `felhom-data` folder into this guest (read-write); the rest of the drive stays hidden |
|
||||
| `GET /guest/memory` | this guest's RAM and its bounds |
|
||||
| `POST /guest/memory` | a bounded RAM resize, applied live (R-24) |
|
||||
| `POST /netstorage/add` | mounts a NAS share on the host and starts a verify job; a failed verify rolls it back |
|
||||
| `GET /netstorage` | the NAS shares |
|
||||
| `GET /netstorage/verify-status` | the verify job's state |
|
||||
| `POST /netstorage/remove` | removes a NAS share |
|
||||
| `POST /controller/swap` | swaps the controller image (202, then async); the agent verifies and rolls back |
|
||||
| `GET /controller/swap/status` | the swap's state |
|
||||
| `POST /escrow/stage-secret` | stages the off-site repository password for the escrow ceremony (0600, never logged) |
|
||||
| `DELETE /escrow/stage-secret` | wipes the staged password |
|
||||
| `POST /escrow/recover-offsite-password` | with the recovery code in the body, returns only the off-site repository password (§8a) |
|
||||
| `GET /escrow/preflight` | the ceremony's checklist |
|
||||
| `POST /escrow/ceremony` | starts the root ceremony job |
|
||||
| `GET /escrow/ceremony/status` | the ceremony's state |
|
||||
| `POST /escrow/ceremony/claim` | the one-shot hand-off of the new recovery code; a later claim is 410 |
|
||||
| `POST /escrow/ceremony/confirm` | the household's two words; only then does the package reach the hub (R-1151) |
|
||||
| `GET /debug/logs` | the agent's debug log ring, for the controller's Debug page |
|
||||
|
||||
### 6a. Implementation (slice 8A — implemented)
|
||||
|
||||
**Status: implemented** (agent v0.10.0 `internal/localapi`; controller v0.35.0 `internal/bootstrap`
|
||||
@@ -314,7 +373,8 @@ token → guest and authorizes per guest, so a compromised controller's blast ra
|
||||
quiesce-on-due consumer is 8B), the rest wrap the existing slice-5/6/7 machinery.
|
||||
|
||||
- **Transport / pin.** The agent serves a **persisted self-signed leaf** bound to the host bridge IP
|
||||
on a fixed port (default `:8443`). The controller pins the **leaf-cert SHA-256** (decision:
|
||||
on a fixed port — `local_api.listen_addr` is REQUIRED, there is no default (`internal/config/config.go:317-319`); an
|
||||
island install (R-50) listens on `169.254.253.1:8443` on its private bridge (corrected 2026-10-24). The controller pins the **leaf-cert SHA-256** (decision:
|
||||
consistency with the agent's Proxmox/PBS cert pinning), carried in its bootstrap. The leaf is
|
||||
generated **once and persisted**, so its fingerprint is stable across agent restarts (a fresh cert
|
||||
each boot would invalidate every already-issued bootstrap pin). Defense-in-depth: the listener
|
||||
@@ -420,8 +480,8 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
|
||||
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
|
||||
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
|
||||
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
|
||||
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent Unreleased
|
||||
> `56aabc7`):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
|
||||
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent
|
||||
> `56aabc7`, released in v0.161.0 — corrected 2026-10-24):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
|
||||
> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no
|
||||
> root command); any other error is returned unchanged, and a failed forced delete is never forced again.
|
||||
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
|
||||
@@ -677,7 +737,8 @@ the *archive* and the *back half*, **and in identity policy** (below).
|
||||
`swap`, and the agent never set it; on the 8 GB minimum machine (guest 4096 MB) the household's first uploads filled the
|
||||
guest with its swap full, the kernel could evict only program code, and the box froze (memory „full" pressure 96 %, load
|
||||
162, the dashboard dead 6½ minutes). `[DESIGN]` The size step sets `swap` = the guest's memory: at bring-up, at a RAM
|
||||
resize (`POST /guest/memory`), and once a minute after agent start for every pool LXC whose swap is below its RAM (live,
|
||||
resize (`POST /guest/memory`), and ONCE, one minute after agent start, for every pool LXC whose swap is below its RAM (`cmd/felhom-agent/main.go`
|
||||
`EnsureGuestSwap`; „once a minute" corrected 2026-10-24) (live,
|
||||
no restart, never lowers). Proven on a fresh box the night of 2026-10-21 (`audits/night-2026-10-21/P1-soak/`: 512 →
|
||||
4096 one minute after the fixture agent started; the 2-hour household load then answered every 10-second probe). **The
|
||||
host must have swap behind it:** our installer gives an 8 GB / 256 GB machine 7.75 GB (measured the same night); a BYO
|
||||
|
||||
@@ -15,9 +15,9 @@
|
||||
|
||||
|
||||
|
||||
> Status: design draft (decision content), grounded on `docs/tests/phase4-signing-findings.md`.
|
||||
> Status: design draft (decision content), grounded on `documentation/tests/phase4-signing-findings.md`.
|
||||
> To be reviewed by Claude Code against that spike + `03` §4, then placed at
|
||||
> `docs/architecture/04-control-plane-authorization.md`.
|
||||
> `documentation/architecture/04-control-plane-authorization.md` — where it is now (paths corrected 2026-10-24).
|
||||
>
|
||||
> Builds on Part 1 (enrollment / trust), Part 3 (the agent verifies + the §4 reversibility gate).
|
||||
> This doc defines the **mechanism** behind `03` §4's "an operator signature the hub can't forge."
|
||||
@@ -75,7 +75,12 @@ recorded last**, only after everything else passes, so an invalid signature can
|
||||
nonce (DoS-safe). Each layer is mandatory and was proven to reject independently (Phase 4 §3–4):
|
||||
- **target binding** — `target.host_id`/`guest_id` must equal *this* box/guest (a signature for box
|
||||
A cannot be replayed at box B);
|
||||
- **time window** — `now ∈ [issued_at, expires_at]`;
|
||||
- **time window** — `now ∈ [issued_at, expires_at]`; as built, `issued_at` may be up to **2 minutes** in the future
|
||||
(clock skew), `expires_at` is never extended — the agent (`felhom-agent/internal/authz/verifier.go:19-22,148-155`) and
|
||||
the root wrapper (`configs/felhom-os-apply`) alike (corrected 2026-10-24). **[DESIGN — the operator picked a code change
|
||||
2026-10-25 (D44); row R-1280]:** neither caps the window's LENGTH (`expires_at − issued_at`); `felhom-opsign` defaults to 30 min
|
||||
(`cmd/felhom-opsign/main.go:57`), and the nonce store keeps a record until its expiry. The pick: the agent and the
|
||||
root wrapper refuse a window longer than 24 h. Not built yet;
|
||||
- **nonce** — unseen within the window (the nonce store **must be persistent across agent restarts**
|
||||
and expiry-pruned; a non-persistent store reopens the replay window after a restart).
|
||||
|
||||
@@ -166,7 +171,7 @@ op the agent verifies** (same pipeline, §2.3) — never unauthenticated config.
|
||||
queue — then the agent polls, verifies, executes, and audits. One command + passphrase, from the
|
||||
desk. **Never** a site visit.
|
||||
|
||||
### 6.1 Operator actions in the report reply [DESIGN, R-314/R-279 — `09` §3 decision 185, controller + hub main 2026-10-08, unreleased]
|
||||
### 6.1 Operator actions in the report reply [DESIGN, R-314/R-279 — `09` §3 decision 185; released in controller v0.304.0 + hub v0.144.0 — corrected 2026-10-24]
|
||||
|
||||
Routine, non-destructive operator requests need no signature (`03` §4 asks for one only to destroy or overwrite the
|
||||
only copy). The hub stores a row, bumps the box's intent, and the report reply carries `operator_actions:[{id,action,arg}]`
|
||||
|
||||
@@ -22,8 +22,9 @@
|
||||
> The hub is **not** greenfield — it's a mature service (felhom-hub **v0.11.0** as of 2026-06-13, Go +
|
||||
> SQLite on k3s, `hub.felhom.eu`). This doc is the **deltas** to evolve it for the Proxmox model, plus
|
||||
> the new data model. Builds on Part 1 (trust/enrollment), Part 3 (the agent + reconcile), Part 4 (signing).
|
||||
> NOTE: this remains a design-draft for the deltas; the live hub is v0.11.0 (the prior "v0.6.3" reference
|
||||
> was stale). Ground specifics against `felhom.eu/hub/` before relying on them.
|
||||
> NOTE: this remains a design-draft for the deltas. It was written against hub v0.11.0 (2026-06-13); the deployed
|
||||
> version is read from `manifests/hub.yaml`, never from this line (corrected 2026-10-24, review round 2: the old
|
||||
> "the live hub is v0.11.0" was a current-state version literal). Ground specifics against `felhom.eu/hub/` before relying on them.
|
||||
|
||||
## 1. Source-of-truth model — two drivers, two directions
|
||||
|
||||
@@ -91,7 +92,7 @@ Evolves the existing staleness checker (60s **cadence**, a **configured** thresh
|
||||
|
||||
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
|
||||
liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known
|
||||
silence (R-1001, hub 0.162.0 — built 2026-10-19, not yet deployed):** a host whose newest `host_stale`/`host_down`
|
||||
silence (R-1001, hub 0.162.0 — deployed since; corrected 2026-10-24):** a host whose newest `host_stale`/`host_down`
|
||||
event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge
|
||||
from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed
|
||||
lookup lets the alarm through.
|
||||
@@ -139,7 +140,7 @@ itself to the floor (R-1052: not yet proven end to end). Setting the global floo
|
||||
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
|
||||
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
|
||||
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
|
||||
offers real agent packages only (R-1011, hub 0.162.0, not yet deployed):** the sha lookup asks for exactly the
|
||||
offers real agent packages only (R-1011, hub 0.162.0, deployed since; corrected 2026-10-24):** the sha lookup asks for exactly the
|
||||
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
|
||||
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
|
||||
(`artifact_file_missing`).
|
||||
@@ -295,6 +296,12 @@ What remains:
|
||||
PBS repo and namespace in the **DR recipe** (`dr_recipe`, reported every cycle — R-106 fixed it to match the
|
||||
backup); the endpoint's PBS fingerprint in **tenantsync** (`hub/internal/tenantsync`); the wrapped key in
|
||||
**`host_escrow.blob`** (and the identity blob beside it).
|
||||
**[DESIGN] 2026-10-24 (R-1254; committed felhom.eu `dcf4f84c`, hub next v0.170.0 — NOT deployed):** the escrow PUT
|
||||
stores the blob, its sealed-password hash and the identity blob in ONE transaction (`Store.SaveHostEscrowWithIdentity`;
|
||||
the superseded copy is made inside it). A failure leaves all of the old package or all of the new — never the new hash
|
||||
beside the old identity package, which a box would read as „escrowed" while the written code opened nothing. A
|
||||
malformed identity blob is refused before anything is stored. (The agent ceremony's „the previous code still works"
|
||||
after a lost answer, R-1254's other half, stays open.)
|
||||
|
||||
Both existing infra-backup tables retire — `infra_backup_versions` (the current/live one, all readers
|
||||
hit it) **and** `infra_backups` (the deprecated legacy mirror). (The planned slim DR record on `hosts` was never
|
||||
@@ -326,8 +333,9 @@ customer instead of a single controller.
|
||||
- **DROP `reports` + CREATE `guest_reports`** (under the clean cutover this is drop+create with no data
|
||||
migration, not an in-place rename); `guest_reports` adds `guest_id`, `host_id`; `cpu/memory` mean
|
||||
guest-level; `backup_last_snapshot` goes quiet.
|
||||
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`; the
|
||||
slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`.
|
||||
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`. ~~The
|
||||
slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`~~ — retired, never built (§9
|
||||
above, R-105) *(corrected 2026-10-24, review round 2)*.
|
||||
- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR
|
||||
recipe, tenantsync and the escrow blob replace them (R-105).
|
||||
- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`,
|
||||
@@ -350,7 +358,9 @@ customer instead of a single controller.
|
||||
|
||||
The box's console tells the volunteer to open „az e-mailben kapott link". That mail must exist whenever
|
||||
a customer is **waiting for a box**, without an operator press. `autoMintSelfBindLink` runs on four
|
||||
events: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
|
||||
events (a fifth path mails a link too: the household's „new link" on a USED or EXPIRED link — one mail per customer
|
||||
per hour, R-719; when the customer still has a linked host it shows the „oldbox" page in the customer's language and
|
||||
tells the operator instead, R-1203 — `internal/web/selfbind.go` `handleBindResend`) *(corrected 2026-10-24, review round 2)*: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
|
||||
bound host** (config save); **a host delete** that keeps the customer. The last two re-check that no
|
||||
host is bound, so a customer who has a box is never sent a pairing link. **Appliance registration is
|
||||
not a trigger** — it knows no customer (`api/appliance.go`). Every send is stored as a hub-internal
|
||||
@@ -388,7 +398,7 @@ it has seen any box screen, and until v0.118.0 nothing here described that.
|
||||
|
||||
| Mail | Trigger | Rendered by | Language source |
|
||||
|---|---|---|---|
|
||||
| Event notification (39 event types) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
|
||||
| Event notification (44 event types — the `mail.event.*` keys of `internal/i18n/locales/hu.json`, counted 2026-10-24) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
|
||||
| Claim / reset / re-enroll / claimed | the claim arc | `FormatClaimEmail` | created-with (no box has reported yet) |
|
||||
| Self-bind link | customer creation, or the operator's button | `FormatSelfBindEmail` | created-with |
|
||||
| The public bind PAGE at `/bind/<token>` | the customer opens the link | one template per language | created-with, **except `expired`** — see 15.4 |
|
||||
@@ -456,7 +466,7 @@ The operator mail renders the counts as one `Repair:` line. The household's side
|
||||
date taken) are shown on its own app page, on its box, until it dismisses them — file names never leave the box.
|
||||
|
||||
|
||||
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, built, NOT deployed; R-1204]
|
||||
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, deployed with 0.167.0; R-1204 — corrected 2026-10-24]
|
||||
|
||||
**[FACT, hub `e267c5e2`]** The host report carries the agent's `wireguard` block (`hostWireguard`). `checkWGTunnel` (`api/host_tunnel.go`, called after `checkHostSwap`) calls the tunnel stale when the box reports `handshake_future_s` > 0 (the clock-ahead lock-out) or a last handshake older than 6 h (healthy boxes measured at most 113–124 s over 24 h). It is edge-triggered against the host's latest tunnel event (`store.LatestHostTunnelEvent`): `wg_tunnel_stale` once on the way down, `wg_tunnel_ok` once on the way back; both are allow-listed and operator-only (`notify.operatorOnlyEvents`, `wg_tunnel_operator_only_test.go`) — the household cannot repair a tunnel. A box with no `wireguard` block (an agent before the field, or no tunnel) raises nothing.
|
||||
|
||||
@@ -555,14 +565,38 @@ record unreadable: its serve paths answer 500, a save refuses it (never blanks i
|
||||
to a hub before v0.138.0** needs the columns opened first: `felhom-hub -unseal-box-secrets` (all or nothing; exit 1 =
|
||||
nothing changed) — run in the running pod, then put the old image back. `guests.api_key` is an unused, always-empty
|
||||
column and is not sealed. *Decided by CC unattended — operator may reverse (`09` §3 decision 132).*
|
||||
**Since v0.168.0 (R-975, R-1241) the last plain tokens are sealed too:** the Cloudflare tunnel token, the Cloudflare API
|
||||
token and the git token live INSIDE `customer_configs.config_json` (`infrastructure.cf_tunnel_token`,
|
||||
`infrastructure.cf_api_token`, `git.token`). Each of those JSON leaves holds the same `enc:v1:` seal; the rest of the JSON
|
||||
stays readable. `SaveCustomerConfig` seals, every reader opens through `openCustomerSecrets` (a leaf that does not open is
|
||||
blanked and marks the record unreadable, as above). Legacy rows are sealed at start-up (`SealLegacyConfigTokens`, without
|
||||
a `config_version` bump — the box's config is byte-for-byte the same, so no box re-pulls or restarts), followed ONCE by
|
||||
`VACUUM` + a WAL checkpoint, because an `UPDATE` leaves the old plaintext row in a freed block of the file
|
||||
(`TestR975_DBFileWithoutKeyRevealsNoToken`). `-unseal-box-secrets` opens these leaves too, so the roll-back above also
|
||||
covers a return to a hub before v0.168.0. The full list of secret-bearing columns, measured on a copy of the live database
|
||||
2026-10-23: `audits/hub-token-seal-2026-10-23/README.md`.
|
||||
**Deployed 2026-10-24 — and what the live file still holds (operator ruling 2026-10-24, accepted).** On the live hub the
|
||||
start-up `VACUUM` did not finish: the Longhorn disk writes ~1.3 MB/s (the 6-hourly `VACUUM INTO` of 530 MB takes ~7 min)
|
||||
and the liveness probe stopped the first start after ~90 s; the second start found nothing left to seal (R-1249). So the
|
||||
rows are sealed, but the live `hub.db` and its `-wal` still carry the old token bytes in freed space (8 of 8, measured).
|
||||
**Every copy that leaves the volume is clean**: the hub's snapshot and the ep0 copy are written by `VACUUM INTO` (the ep0
|
||||
copy of 06:30Z: 0 of 8), and the volume is out of DooPlex's Longhorn backup (D40). Accepted, not cleaned: whoever can read
|
||||
the live volume is root on DooPlex and can read `OFFSITE_SECRET_KEY` there too; a full wipe would also need a hub stop of
|
||||
8–20 minutes, a disk trim and the volume's Longhorn snapshots deleted. **Rule: a hand copy of `hub.db` for analysis is
|
||||
made with `VACUUM INTO` (or taken from ep0), never `cp`/`tar` of the live file** — a `tar` of the file carries the old
|
||||
bytes (`audits/line-test-2026-10-24/hub-0168-seal-check.txt`).
|
||||
**And the key is the other half:** a backup of the database restores a hub that can open the sealed columns only
|
||||
with the same `OFFSITE_SECRET_KEY`; today that key exists only on DooPlex (the k8s Secret, and the GPG secrets export
|
||||
on the same machine). The off-site plan for the database and its key: `runbooks/RUNBOOK-hub-db-offsite-backup.md`
|
||||
(R-173 — option A decided 2026-10-05, `09` decision 125; §16.3).
|
||||
|
||||
### 16.3 The nightly database snapshot (R-173, hub v0.136.0)
|
||||
### 16.3 The database snapshot — every 6 hours since D34 (R-173, hub v0.136.0; D34 in hub 0.167.0)
|
||||
|
||||
Every night at **02:00 Budapest** the hub writes `<data>/snapshots/hub-<UTC stamp>.db` with SQLite's `VACUUM INTO`:
|
||||
**Corrected 2026-10-24 (review round 2):** since D34 (operator yes 2026-10-23) the hub writes a snapshot at **02:00,
|
||||
08:00, 14:00 and 20:00 Budapest** (`internal/dbsnap/dbsnap.go` `Slots`), catches up at start when the newest is older
|
||||
than **6 h** (`CatchUpAge`), and DooPlex pushes the newest at **:30 after each slot** (`felhom-hub-db-backup.timer`),
|
||||
refusing one older than **7 h** (`felhom-hub-db-backup` `MAX_AGE_H`). The paragraph below is the original nightly
|
||||
design; its mechanics hold per slot. Originally: every night at **02:00 Budapest** the hub writes `<data>/snapshots/hub-<UTC stamp>.db` with SQLite's `VACUUM INTO`:
|
||||
one statement, one point in time, every committed write included — the rows still in `hub.db-wal` too, which a file copy
|
||||
of `hub.db` loses (measured in `internal/dbsnap` tests: a plain copy held 0 of 150 fresh rows). Written as `.tmp`, mode
|
||||
`0600`, then renamed, so a reader never sees half a file. The newest **2** stay (the volume grew from 1 GiB to 2 GiB
|
||||
@@ -591,7 +625,7 @@ failure never fails or delays a request: lines go through a non-blocking buffer
|
||||
|
||||
### 16.5 Daily jobs and the clock change [DESIGN — hub v0.163.0, operator ruling D24 2026-10-21, R-1183]
|
||||
|
||||
Every daily hub job (`scheduleDaily`: db-snapshot 02:00, prune, deadline-check 05:00, offsite-key-audit 07:10) runs at
|
||||
Every daily hub job (`scheduleDaily`: the four db-snapshot slots 02:00/08:00/14:00/20:00, prune at its configured time, deadline-check 05:00, offsite-key-audit 07:10 — slots corrected 2026-10-24) runs at
|
||||
its Budapest wall-clock time **once per calendar day**: the next run is the calendar day after (`time.Date`, day+1),
|
||||
never „now + 24 h", and the slot that ran last never runs again. On a 25-hour day (the last Sunday of October) the old
|
||||
arithmetic ran every job after 02:00 twice; on a 23-hour day it ran them an hour late. Pinned by `TestR1183_*` for
|
||||
|
||||
@@ -19,7 +19,8 @@
|
||||
> decisions; grounded against felhom.eu @ `bf099f6` and felhom-agent @ `4ba1b14` (v0.63.0).
|
||||
> Evidence base: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (all
|
||||
> measurements cited below as "spike Pn"). This document ends in an implementation slice roadmap
|
||||
> (§8); **no slice is implemented yet** — each is a separate future task. Consistent with and
|
||||
> (§8); **no slice is implemented yet** — each is a separate future task. *(Dated note 2026-10-24, review round 2: true when
|
||||
> written; S1–S4 and S5's safe halves have since shipped — §8 and the agent's `internal/wgtunnel` say what is built.)* Consistent with and
|
||||
> extending Part 01 (`01-topology-and-trust.md`); where this doc names a component's duty it uses
|
||||
> Part 01 §3 vocabulary.
|
||||
|
||||
@@ -140,7 +141,9 @@ before this ships (the drilled `--rescope-acl` posture is unaffected).
|
||||
4. **Hub → endpoint VM:** adds `{pubkey, /32}` to the WG server peer list (slice-1 surface).
|
||||
5. **Box (agent):** next heartbeat sees the generation advance → fetches desired-state → writes
|
||||
the WG config → `install` + `systemctl enable --now wg-felhom` (the sudoers `*.mount` pattern
|
||||
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3).
|
||||
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3). *(As built since R-861: no `install` sudoers
|
||||
line; the conf is installed 0600 to its fixed path by the root wrapper `felhom-priv-apply wg` —
|
||||
`felhom-agent/configs/felhom-agent.sudoers:223-229`; corrected 2026-10-24.)*
|
||||
6. **Box:** tunnel handshakes (outbound, keepalive); PBS storage is (re)pointed at
|
||||
`pbs_tunnel_ip:8007` (§3.4); first backup runs.
|
||||
|
||||
@@ -169,7 +172,8 @@ tenant ACL (`DatastoreBackup` on `/datastore/<ds>/<ns>` to BOTH the user and the
|
||||
### 3.5 Key custody + DR re-establishment
|
||||
|
||||
The WG private key joins the **escrowed IdentityBundle**
|
||||
(`internal/escrow/identity.go:24-27`, today `{TunnelToken, PBSToken}` → `+ WGPrivateKey`) — the
|
||||
(`internal/escrow/identity.go`, then `{TunnelToken, PBSToken}` → `+ WGPrivateKey`; as built the bundle carries FOUR
|
||||
fields — `TunnelToken`, `PBSToken`, `WGPrivateKey`, `ResticRepoPassword` — `identity.go:30-40`, corrected 2026-10-24) — the
|
||||
same R-wrapped, zero-knowledge age blob the DR recipe already consumes. On host-loss
|
||||
re-provisioning (01 §9 restore mode): the box recovers the bundle with the customer's recovery
|
||||
code, restores the WG key, and re-establishes the tunnel; hub-side the peer entry still exists
|
||||
@@ -199,7 +203,8 @@ disables the service, KEEPS its marker, and never re-registers on its own — re
|
||||
revoked until the operator re-adds the peer (the heartbeat report carries the box's pubkey as
|
||||
the recovery handle; **re-add via `POST /hosts/{id}/wg` with the global key** — the raw registry
|
||||
add does not bump the host's generation). The only automatic re-registration is the
|
||||
pubkey-mismatch (DR/re-key) path, backoff-bounded. Absent DATA (a failed fetch) is never a
|
||||
pubkey-mismatch (DR/re-key) path, backoff-bounded — and, since D30 (agent v0.163.0), a key the agent makes itself after
|
||||
a handshake stamped in the future for 15 min (`internal/wgtunnel/rekey.go`; corrected 2026-10-24). Absent DATA (a failed fetch) is never a
|
||||
teardown signal. **Rollout gate:** `wg_tunnel.enabled` defaults FALSE agent-side until the
|
||||
production endpoint exists.
|
||||
|
||||
@@ -285,7 +290,9 @@ production endpoint exists.
|
||||
`wg-felhom` AllowedIPs so it survives the agent's self-heal ([OF-1]); PBS unaffected.
|
||||
- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one: no handshake
|
||||
within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path
|
||||
carries it. No new alarm channel.
|
||||
carries it. No new alarm channel. *(Since hub 0.166.0 there is one: `wg_tunnel_stale` / `wg_tunnel_ok`,
|
||||
operator-only — a handshake older than 6 h or stamped in the future, D30 / R-1204, `hub/internal/api/host_tunnel.go`;
|
||||
corrected 2026-10-24.)*
|
||||
**EXTENDED 2026-07-05 (TASK H1).** The agent's heartbeat now also carries an **`oob` stanza**
|
||||
(`felhom_sshd_active`, `felhom_sshd_port`, `reachable`, `config_invalid`, `operator_peer_configured`,
|
||||
`operator_key_configured`, `wg_handshake_age_s`) — the operator's "can I get into this box right
|
||||
@@ -297,6 +304,11 @@ production endpoint exists.
|
||||
|
||||
## 5. The endpoint VM (the one new managed-infra piece)
|
||||
|
||||
**AMENDED 2026-10-24 (R-1247, operator ruling):** a fourth, small thing — `felhom-iperf3.service`, an iperf3 server bound to
|
||||
`10.77.0.1:5201` and opened in nftables for `wg0` from `10.77.0.0/24` only, so each box's agent can measure its upload
|
||||
line once a week (felhom.eu `scripts/ep0-iperf3/`; agent `internal/linetest`). It never answers on the internet (proven
|
||||
2026-10-24, `audits/line-test-2026-10-24/`).
|
||||
|
||||
Runs three things, nothing else: the WG server (peer list = hub-written), PBS (one datastore,
|
||||
per-customer namespaces), and the minimal peer-sync surface the hub drives. **How** the hub
|
||||
mutates the peer list — a tiny authenticated push API on the VM vs the VM pulling a signed peer
|
||||
@@ -318,12 +330,16 @@ traverse `/root`) — create datastores under a world-traversable parent (`/srv`
|
||||
## 6. Trust-model conformance (01 §3/§4 preserved)
|
||||
|
||||
- The box **only ever dials out** (UDP to the endpoint; the same posture as hub-poll and
|
||||
PBS-push, 01 §4/§7). No inbound listener is added to any customer box; the hub still never
|
||||
initiates into a box.
|
||||
PBS-push, 01 §4/§7). ~~No inbound listener is added to any customer box; the hub still never
|
||||
initiates into a box.~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** Two listeners sit on the box's tunnel side, reachable only through
|
||||
the tunnel: the agent's poke listener (UDP 51822 on the box's WireGuard address — a poke carries no command, the box
|
||||
still pulls) and, on an appliance install, the operator door `felhom-sshd` (only from the operator's `/32`, §4.5) —
|
||||
the documented break-glass (`03` §5). The hub still never sends a command into a box.
|
||||
- The endpoint VM cannot reach into boxes either: it terminates tunnels whose per-peer routes
|
||||
(§4.5) admit only box→PBS traffic; it holds no box credentials.
|
||||
- **Out of scope, explicitly:** (a) operator-inbound access to boxes — a *separate* future
|
||||
problem with its own trust decision; nothing here creates or presumes it; (b) game-server
|
||||
- **Out of scope, explicitly:** (a) ~~operator-inbound access to boxes — a *separate* future
|
||||
problem with its own trust decision; nothing here creates or presumes it;~~ operator-inbound access to boxes was
|
||||
added on 2026-07-05 (TASK H1) and rides this tunnel — §4.5 (corrected 2026-10-25, D45, R-1266); (b) game-server
|
||||
ingress — public players are not tunnel peers; that is a different ingress problem (public
|
||||
reachability), not a backup-transport one.
|
||||
|
||||
@@ -350,7 +366,8 @@ venue; it gates nothing. Runbook:
|
||||
**Open sub-decisions (deferred by design):**
|
||||
- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — **slice-1 design point** (§5).
|
||||
- WG key rotation cadence/mechanics — the reconcile channel can carry it (new pubkey → hub →
|
||||
endpoint swap); design when fleet size warrants, possibly the moment the "separable fleet
|
||||
endpoint swap); design when fleet size warrants *(one automatic rotation exists since D30: after a future-stamped
|
||||
handshake for 15 min, at most once an hour — `rekey.go`; a scheduled rotation is still open; noted 2026-10-24)*, possibly the moment the "separable fleet
|
||||
layer" (D1) conversation reopens.
|
||||
- Endpoint scaling/sharding (N customers per VM, second region, HU-hosted option) — capacity
|
||||
planning, not architecture; the peer registry already keys per-host.
|
||||
@@ -373,6 +390,8 @@ venue; it gates nothing. Runbook:
|
||||
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
|
||||
|
||||
**MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6
|
||||
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a
|
||||
read-only prober that names itself "the seam for the tunnel-management slice") adopts the
|
||||
S3-piloted agent-managed-service pattern in a later, separate slice.
|
||||
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — then a
|
||||
read-only prober that named itself "the seam for the tunnel-management slice") ~~adopts the
|
||||
S3-piloted agent-managed-service pattern in a later, separate slice.~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** This plan is withdrawn:
|
||||
cloudflared is a GUEST container that the controller renders and keeps up; the agent only reads it (four states,
|
||||
`cloudflared.go:12-23`) and will not manage it — `03` §2/§3 wins.
|
||||
@@ -253,7 +253,7 @@ It carries guest sizing, `pve_storage[]`, and an app half with per-app `storage_
|
||||
the identity bundle's shape is `{tunnel_token, pbs_token, wg_private_key, restic_repo_password}`
|
||||
(`felhom-agent/internal/escrow/identity.go:26-39`).
|
||||
|
||||
**[DESIGN] Off-site deletion custody (decisions 68–69, 2026-10-03) — BUILT hub v0.127.0 / controller v0.289.1, live on both demo boxes.** The restic repository password stays on the box only. The box's off-site key is append-only (pinned in `authorized_keys`, written by the hub — the key registrar); the box never receives the sub-account password, which the hub stores encrypted at rest. Old snapshots are pruned by the box itself, only in a weekly window the hub opens, behind a fake-snapshot guard (R-822); the window has run live on both demo boxes since 2026-10-05 (R-95, closed). The guard's residual — past-dated fakes steering the keeps, and a window check that trusts the box's own counts — is R-822 and R-895. **Hub v0.158.0 (built 2026-10-18, deploy waits on the operator) adds the hub's own check:** the 07:10 key audit compares the repository's `snapshots/` listing, over the hub's own login, with yesterday's — files gone that no clean-up window explains raise `offsite_snapshots_vanished` (error), files no reported box run explains raise `offsite_snapshots_unexplained` (warning); both operator-only (R-895 slice 2). A Felhom-side pruner holding repository passwords is rejected.
|
||||
**[DESIGN] Off-site deletion custody (decisions 68–69, 2026-10-03) — BUILT hub v0.127.0 / controller v0.289.1, live on both demo boxes.** The restic repository password stays on the box only. The box's off-site key is append-only (pinned in `authorized_keys`, written by the hub — the key registrar); the box never receives the sub-account password, which the hub stores encrypted at rest. Old snapshots are pruned by the box itself, only in a weekly window the hub opens, behind a fake-snapshot guard (R-822); the window has run live on both demo boxes since 2026-10-05 (R-95, closed). The guard's residual — past-dated fakes steering the keeps, and a window check that trusts the box's own counts — is R-822 and R-895. **Hub v0.158.0 (built 2026-10-18; deployed the same day on the operator's yes — corrected 2026-10-24) adds the hub's own check:** the 07:10 key audit compares the repository's `snapshots/` listing, over the hub's own login, with yesterday's — files gone that no clean-up window explains raise `offsite_snapshots_vanished` (error), files no reported box run explains raise `offsite_snapshots_unexplained` (warning); both operator-only (R-895 slice 2). A Felhom-side pruner holding repository passwords is rejected.
|
||||
|
||||
**[DESIGN, built hub 2026-10-07 — R-32 option A, `09` §3 decision 167] What RESET does to the household's off-site
|
||||
copy.** On the shared pool box a sub-account is a LOGIN, not the data: deleting it leaves its folder (measured
|
||||
@@ -380,7 +380,7 @@ never moved by an update or an undo, and the unit restore accepts it.
|
||||
|
||||
| Tier | Location | Captures | Cadence (LIVE) | Retention (LIVE) | Encrypted |
|
||||
|---|---|---|---|---|---|
|
||||
| **Tier-1** recovery unit | `<nsRoot>/backups/primary/<app>/` on the app's own drive | compose + `.felhom.yml` + **secret-stripped** `app.yaml`, `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no |
|
||||
| **Tier-1** recovery unit | `<nsRoot>/backups/primary/<app>/` on the app's own drive | compose + `.felhom.yml` + `app.yaml` (0600 — since D5 / schema 2 it carries the PORTABLE secrets, `recovery_unit.go:25`; „secret-stripped" corrected 2026-10-24), `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no |
|
||||
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
|
||||
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
|
||||
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/felhom` (its `docker/` and `sys_drive/` folders, bound in the guest to `/var/lib/docker` and `/mnt/sys_drive` — §7.5.1; `mp1` no longer exists, corrected 2026-10-15) | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` → `--prune-backups keep-daily=3`: one copy per day, 3 days (keep-last until agent 0.160.0, R-1149) | **no** |
|
||||
@@ -437,7 +437,7 @@ container must keep its id). Ring 1 never takes it in the night leg. Measured: t
|
||||
|
||||
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
|
||||
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
|
||||
(`cmd/controller/main.go:604-607`). Both boxes run W = `02:30`.
|
||||
(`backupwindow.LegTimes`, called at `cmd/controller/main.go:1258-1264`; line corrected 2026-10-24). Both boxes run W = `02:30`.
|
||||
|
||||
**[FACT, controller v0.271.0, 2026-09-25] A fourth leg: automatic app updates, CHAINED after the off-site
|
||||
leg** (`09` §3 decisions 11 and 20, §6.4 part 7). The `offbox-backup` job runs the update leg when its off-site
|
||||
@@ -448,8 +448,10 @@ night the whole-guest backup starts later inside its own window; it is never ski
|
||||
numbers: `audits/DRILL-night-2026-09-25.md` Part D.
|
||||
|
||||
**[FACT, 2026-09-25] demo-hp's local whole-guest target is its host ROOT disk; retention there is now
|
||||
keep-last=1** (operator ruling 2026-09-24, option A; R-684). PVE prunes AFTER a successful backup, so the target
|
||||
must hold keep-last + 1 archives during the run — measured: a 9201 archive is 8.18 GB, the root disk 39 GB. The
|
||||
keep-last=1** (operator ruling 2026-09-24, option A; R-684) — *since R-1149 (operator D17, 2026-10-19) the local tier's
|
||||
spec is `keep-daily=N` (`felhom-agent/internal/config/config.go:553-559`), so demo-hp is keep-daily=1; corrected
|
||||
2026-10-24*. PVE prunes AFTER a successful backup, so the target
|
||||
must hold N + 1 archives during the run — measured: a 9201 archive is 8.18 GB, the root disk 39 GB. The
|
||||
warning that should precede such a failure is R-685.
|
||||
|
||||
**[FACT, 2026-10-14] A power cut during the whole-guest backup leaves a half-written archive, and from agent v0.156.1 the
|
||||
@@ -731,7 +733,8 @@ definition is written or a named volume replaced — a failed placement, volume
|
||||
the live definition back and HOLDS the app stopped (`restore_mixed`, operator-cleared); the household reads that it
|
||||
needs our help; the operator gets a `backup_run_failures` line. The hold is persisted before the stop. Placed files
|
||||
ALONE (no version change, no volume replaced) still restart after a successful rollback (`TestR379_ScenarioA`): that
|
||||
mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893. **`[FACT]`
|
||||
mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893 *(shipped since:
|
||||
D20 in controller v0.336.0 and D22 in v0.338.0, below — noted 2026-10-24)*. **`[FACT]`
|
||||
2026-10-19 (measured on 9202, controller 0.335.0, `audits/night-2026-10-19/P3/`):** a forced replay failure after the
|
||||
volume was replaced left the NEWEST rows (the undo dump) on top of the backup's volume — a table only the backup had
|
||||
stayed (37 tables, the live app had 36); the definition did not move (same version); the app was held and a start
|
||||
@@ -747,13 +750,17 @@ the snapshot added is removed, the live definition is written back and the app i
|
||||
restore failed and nothing changed (with placed files: its database and volumes are back, the copied files stay); the
|
||||
operator gets `backup_run_failures` leg `restore-put-back`. The hold (`restore_mixed`) remains only for a put-back that
|
||||
fails — then the copy is kept for support — and the double failure (replay AND rollback) holds as before. On success the
|
||||
copy is removed. **`[FACT]` 2026-10-20 (9202, controller 0.336.0-rc.1, `audits/night2-2026-10-20/P2/live/`):** the same
|
||||
copy is removed — **since controller 0.343.0 (fix A, 2026-10-25) only after a STABLE start, and the verdict and the release
|
||||
are two different tests (§8.2e):** the verdict „started" needs the stack running on every sample of a 45 s window, no
|
||||
container restart in it and every healthcheck `healthy`; the copy is released only after a further 3 minutes of the same
|
||||
(10 s samples, watched up to 30 minutes; a new restore of the app ends the watch and leaves the copy to it). A restore that
|
||||
passes the verdict and then flaps keeps its copy. **`[FACT]` 2026-10-20 (9202, controller 0.336.0-rc.1, `audits/night2-2026-10-20/P2/live/`):** the same
|
||||
forced failure as on 2026-10-19 → 36 tables, the backup-only table absent, schema/data/row-count md5 equal to before,
|
||||
the app running, no hold, the copy (69 MB, 0.9 s) gone; a normal restore of the same snapshot then worked. The R-893
|
||||
known limit above is closed for named volumes; placed FILES are still not taken back (the plain-files mix, as ruled).
|
||||
|
||||
**`[DESIGN]` 2026-10-21 — the put-back covers EVERY restore, and a power cut (operator ruling D22 = yes, 2026-10-21
|
||||
20:05; controller `main`, unreleased).** (1) **The local unit restore** — the app's own unit, the second drive's unit,
|
||||
20:05; released in controller v0.338.0 — corrected 2026-10-24).** (1) **The local unit restore** — the app's own unit, the second drive's unit,
|
||||
the second drive's „Teljes visszaállítás" and a kept load all run `RestoreFromRecoveryUnitAtWith` — now takes the same
|
||||
measure → copy → put-back (`offbox_restore_undo.go`, one implementation, no second copy of the logic): the copy must fit
|
||||
beside the unit before anything stops; ANY failure after the stop (volume replay, definition write, DB-only start, DB
|
||||
@@ -775,6 +782,28 @@ than the unit), the app running, the record reads the D22 sentence. Known limit
|
||||
named volume the unit replaces (none in the catalog today — a drive-file app's unit restore is refused by R-538) is not
|
||||
covered by the volume copy.
|
||||
|
||||
**`[DESIGN]` 2026-10-24 — a pending put-back blocks EVERY start (R-1251; review round 2 HUNT-1 H1, H2, H7; committed in
|
||||
controller `bb2809d` + `062f9b4`, unreleased — rides the next controller release).** For a drive-backed app the copy lives on
|
||||
the app's drive, and at start the drive is often not bound yet. A copy that cannot be reached while its drive is not
|
||||
bound is **waiting, never „gone"**: the crash recovery writes a restore hold (`putback_pending`, the copy's folder
|
||||
in it) and starts nothing. The hold is the one store every start path already honours — the household's
|
||||
Start/Restart/Update, the boot recreate, the drive return, the media restart, the boot sweep, the night chain, the
|
||||
update leg — so no path has its own check. When the drive binds, the put-back runs FIRST (the boot recreate and the
|
||||
drive-return restart; the start and every drive return sweep the waiting holds), the hold is lifted, then the app
|
||||
starts. The household reads „…A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el…" on the
|
||||
app and „Ne indítsd újra a visszaállítást…" in the restore record. A put-back that FAILS (at start or when the drive
|
||||
returns), or a copy gone while its drive is bound, leaves the app **held** with the D20 hold („félúton megszakadt"),
|
||||
the copy kept, and the marker lets go of the app — never stopped-and-unheld (H2). Stop pressed during the restore still
|
||||
gets the put-back; only the start is skipped (H7). A removal clears a waiting put-back; a later restore that completes
|
||||
drops it (the household chose that data). The copy's folders are synced to disk BEFORE the marker names the copy.
|
||||
**`[FACT]` 2026-10-24 night, Tester 1 (VM 341; MeTube on the data drive, 1.5 GB in its volume;
|
||||
`audits/restore-holes-2026-10-24/P1-tester1/`):** a local restore cut by powering the VM off 2.1 s after „the restore
|
||||
copy is recorded", the data disk taken off, the box booted: held as waiting, Start and Update refused (409, the
|
||||
sentence); the disk plugged back: the agent re-mounted it, the boot recreate put the copy back BEFORE the start, the
|
||||
household's newest data („gen2") and its download on the drive were there, and a later controller restart poured
|
||||
nothing. Run 1 of the same proof found the journal left as `undo.json.tmp` (the rename lived only in the page cache) —
|
||||
the app was held for support, never started on the cut data; the folder sync above fixed it, run 2 passed.
|
||||
|
||||
**[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.**
|
||||
Recorded here rather than only in a closed register row, because a decision that survives only inside
|
||||
a closed work item is a decision nobody will find.
|
||||
@@ -885,6 +914,8 @@ run makes it longer.
|
||||
|
||||
### 6.5 Kept data — what a removed app leaves on the drive (controller v0.274.0, `09` §3 decision 36)
|
||||
|
||||
*Second-drive copies that are KEPT (D25) and the rule that a rejected backup block never deletes one: §8.2a (R-1252, 2026-10-24).*
|
||||
|
||||
**What it is.** "Remove the app, keep my data" leaves the app's drive folder (`<drive>/appdata/<app>`, or the folder
|
||||
its definition binds through `${HDD_PATH}`) in place. A reinstall over it no longer runs silently into the old files
|
||||
(R-657): the install asks „A megőrzött adataimat használom" / "Use my kept data" (the database from the newest copy of
|
||||
@@ -910,6 +941,26 @@ holds no database, or **does not record its data version** (§6.6 — a unit wri
|
||||
a mixed or mismatched unit is refused by §6.6's own rule. Then a dated folder's files move back and the one unit restore
|
||||
runs. The downloaded copy is removed on every path. A failed download or a refusal leaves the kept files as they were.
|
||||
|
||||
**[FACT] 2026-10-25 — why the „another drive" check let another box's copy through (R-1271; catalog sweep 2026-10-24,
|
||||
Tester4: Paperless-ngx, Nextcloud, Immich).** The check (`KeptCopyAt`, `internal/backup/kept_load.go`) compares the
|
||||
unit's `app.yaml` `HDD_PATH` with the drive the kept files sit on. That is a mount PATH, and every box mounts its first
|
||||
data drive under the same name (`/mnt/felhom-drives/hdd_1`) — so a copy taken of another box's disk matches. It is a
|
||||
path check, not a physical identity, and it stays one: loading an old box's data on a new box is what off-site is for.
|
||||
What chose the wrong copy was the DATE: the rehearsal box's clock had been set to 2026-10-28, its off-site snapshot
|
||||
carried that time, this box had no data time of its own to cap it (`offsiteDataTime`), and „newer than every local
|
||||
copy" took it over the family's own copy of the same evening. Immich's 20 minutes of 404 were the ROUTING, not a half
|
||||
load: the unit brought the old box's `SUBDOMAIN` (`photos`), so the app answered at `photos.<domain>` and the probe
|
||||
asked `immich.<domain>`; its containers were healthy.
|
||||
|
||||
**[DESIGN] 2026-10-25 — a copy dated in the future is never „the newest" (R-1271; brief 2026-10-25 Part 4, the safest
|
||||
reading; controller `bd9f4f7`, unreleased, rides v0.343.0).** A copy whose date lies more than **10 minutes** past this box's clock (`KeptClockSkew`: NTP keeps a box
|
||||
within a second; ten minutes covers a box that synced late after a power cut) is future-dated. It never counts as the
|
||||
newest: the newest copy with a true date wins (a tie goes local); the household's own local copy wins over any
|
||||
off-site copy that is not newer by a true date. A future-dated copy is offered only when no other copy exists — then
|
||||
the question says its date is in the future and does not recommend it (the „use" button is not the primary one). The
|
||||
question always names the copy it will use. An off-site copy is never refused only because another box of the
|
||||
household made it.
|
||||
|
||||
**It is NOT backed up.** No tier captures `<drive>/kept` or a leftover `appdata/<x>` of a removed app; the page says
|
||||
so („Erről nem készül mentés." / "This is not backed up."). What brings it back into an app is the unit it carries
|
||||
(or the app's own unit on the drive), listed per row as „Visszatölthető innen" / "Can be loaded from".
|
||||
@@ -973,6 +1024,7 @@ both. Files written under different pins (a failed leg) make the unit `mixed`.
|
||||
|---|---|---|
|
||||
| own unit (Tier 1), second-drive mirror (Tier 2), kept-data Load | present, `compose/` names the data's pins | the data starts under that definition; if it differs from what ran, the restore pages' first sentence names the backup and the version (the kept-data Load has its own sentence and does not add this one) |
|
||||
| same | present, `compose/` names OTHER pins, or `mixed` | **refused before anything is touched** (`ErrUnitVersionMismatch`): „…ezért a visszaállítás nem indult el. Az alkalmazás érintetlen" |
|
||||
| off-site (Tier 3) | present, `mixed`, or a definition that does not belong to its data | **refused before anything is touched** like the rows above (`unitVersionCheck`, `offbox_reconstitute.go:826-833`; row added 2026-10-24) |
|
||||
| off-site (Tier 3) | present, the snapshot's version differs from what runs | the snapshot unit's definition is written into the stack dir (and pinned) right after the stop, before any file, volume or database is touched; the database service is resolved from it |
|
||||
| any | absent — a unit or snapshot written before v0.275.0, or with an unstamped file | restores **as before** (its captured definition; the off-site path the live one), with a WARN naming it |
|
||||
|
||||
@@ -999,13 +1051,28 @@ requires a database dump written after the conversion whose recorded engine is t
|
||||
image is re-pulled on restore. A version its maker has deleted cannot start. Measured 2026-09-26: the 42 ladder
|
||||
digests all resolve today (`audits/version-travel-2026-09-26/A7/`). **Accepted as a known limit — operator ruling
|
||||
2026-10-12 (S8, `09` §3 decision 203; R-698 closed).** What a household meets: a restore whose version the maker has
|
||||
deleted stops at the image pull, the app stays as it was, and the household uses another copy or a newer version.
|
||||
deleted stops at the image pull, the app stays as it was (since D48 by ORDER — the pull comes before the stop; before it,
|
||||
by the D22 put-back), and the household uses another copy or a newer version.
|
||||
The operator's follow-up — *„will need to mirror the apps we support later, just to be sure"* — is **R-933**: mirror
|
||||
the images of every catalog app we support into our own registry, before scale. Until it is built, this limit holds.
|
||||
**Since decision 53 (`09` §3, 2026-09-30 evening):** a box keeps only the image each app service runs and the one before
|
||||
it; older images of the app are deleted. A restore that needs an older version re-pulls it — as before; the limit
|
||||
above is unchanged, and kept data (decision 40) is not touched by the image clean-up.
|
||||
|
||||
**[DESIGN] 2026-10-25 — a restore downloads first (operator ruling D48, `09` §3 decision 257; controller `2d71270`,
|
||||
unreleased — rides v0.343.0).** „The app stays as it was" in the paragraph above is now true BY ORDER, not by the D22
|
||||
put-back. After the existing refusals and before the first stop or change, every restore path resolves every image the
|
||||
definition that will RUN names — present on the box, or pulled now — under the image-work lock held from the first
|
||||
check until the restore's last start (R-863: the image clean-up cannot delete a pulled image before the start). A pull
|
||||
that fails is a REFUSAL: the app is not stopped, its data is not moved, and the household reads which version, and
|
||||
why: „Ennek a mentésnek a verzióját (%s) nem lehet letölteni: a készítője már nem kínálja…" when Docker says the version
|
||||
does not exist (`manifest unknown`, `not found`, `pull access denied`), „…most nem lehet letölteni… Próbáld újra
|
||||
később…" for everything else (a busy or unreachable registry). The operator gets one `backup_run_failures` line (leg
|
||||
`restore-image-unavailable`). The paths: the local unit restore — and with it the second drive's unit restore, the
|
||||
second drive's whole restore (its `CheckOnly` pass carries the pull, so its file half never runs first), a kept-data
|
||||
load from a unit and from off-site, and a set-aside unit (D47) — the off-site restore, the second drive's file restore
|
||||
and the no-unit fallback. Not covered: the `.fab` import — it removes the running app's volumes before its definition is pulled (R-1279).
|
||||
|
||||
### 6.7 Why an off-site run failed — the failure classifier (R-571)
|
||||
|
||||
**[FACT, read from source 2026-10-05, felhom-controller `114ff27`]** When an off-site (restic) run fails, the
|
||||
@@ -1341,12 +1408,13 @@ Added 2026-08-02 from `audits/SPIKE-recovery-unit-space-2026-08-02.md` and
|
||||
which is true and is the whole story for a drive-resident app. **It is not the whole story for an app
|
||||
with no data drive, and that case was undocumented until now.**
|
||||
|
||||
**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go:245-255`)
|
||||
**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go`, now near line 441)
|
||||
returns the app's `HDD_PATH` if it has one and **`systemDataPath` otherwise** — what
|
||||
`internal/appbackup/paths.go:26-27` calls *"the SSD-only system-data fallback"*. Nothing deletes a unit
|
||||
after it is copied onward: the only prune is F5 (`backup.go:1053-1112`), which removes residue on an
|
||||
after it is copied onward: the only prune is F5 (`pruneStalePrimaryDirs`, `backup.go`, now near line 1514), which removes residue on an
|
||||
**old** drive when an app **moves**. So for every app without a data drive, `mp1` holds the kept copy
|
||||
indefinitely.
|
||||
indefinitely. *(Dated note 2026-10-24: `mp1` no longer exists — §6.1 / line 386; the system data now lives under the
|
||||
one `mp0` volume's `sys_drive/` folder, `felhom-agent/configs/build-golden.sh`.)*
|
||||
|
||||
**What a unit contains, which bounds the problem** (`internal/backup/recovery_unit.go:20-25`):
|
||||
`compose/` + `db-dumps/` + `volume-dumps/` (named-volume tars) + `manifest.json`. **It never contains
|
||||
@@ -1495,7 +1563,10 @@ crosses the line — **R-158**.
|
||||
> Eleven rows carry a blank RTO — 4, 5, 8, 9, 10, 11, 11b, 12, 13, 14, 15 — and only six of those are
|
||||
> deferred work. **Row 5** has no recovery to time (a derived copy), **row 13** has no route by
|
||||
> design, **row 14** is proven and merely never stopwatched, **row 11b** is a note not a row, and
|
||||
> **row 15 is an open DEFECT (R-104) that this line does NOT cover.**
|
||||
> **row 15 was an open DEFECT (R-104) that this line did NOT cover** — *R-104 was CLOSED 2026-10-05 (controller
|
||||
v0.297.0: a lock that survives the self-heal gets the class `locked`, its own note and a retry, `offbox.go:156-162`;
|
||||
CLOSED-ITEMS.md; corrected 2026-10-24).* **Row 16** (added 2026-10-25, D44 #6, R-1265 — after this banner) has a blank RTO
|
||||
too: a registry rebuild from the ep0 release copy was never timed.
|
||||
>
|
||||
> **NO STATUS MOVED ON THE DAY THIS WAS WRITTEN, because nothing was proven that day.** A stopping
|
||||
> line that promotes a row is a stopping line that lies. Full text, and the conditions that reopen
|
||||
@@ -1528,13 +1599,14 @@ crosses the line — **R-158**.
|
||||
| 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human |
|
||||
| 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** **`[BETA-DEFERRED]`** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) |
|
||||
| 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** **`[BETA-DEFERRED]`** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) **`[FACT]` 2026-10-03 (decision 70): ep0's datastore now has a nightly copy on DooPlex** (PBS pull-sync `ep0-felhom-offsite`, `remove-vanished false`, weekly verify, failures mailed to the operator); first pull 201 s, 12 GB, 4 of 4 snapshots, matching ep0 namespace for namespace. Restore route: `runbooks/ep0-datastore-copy.md` — never walked. `audits/offsite-lock-build-2026-10-03/partF/`. **`[FACT]` 2026-10-04: the restore route from the DooPlex copy was WALKED** on demo-hp (scratch VMID): listed in 2 s, restored in 186 s (15 GB logical), data read by `pct mount`; the restored config carries `onboot: 1` and the host's real drive binds — strip them on a restore beside the original (R-834). The copy now keeps 8 weekly copies (decision 71). `runbooks/ep0-datastore-copy.md` route 1. |
|
||||
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recoveredLine truncated
|
||||
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
|
||||
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recoveredLine truncated
|
||||
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its encrypted database copy on ep0 (6-hourly, `scripts/hub-db-backup`; **since 2026-10-24 the hub volume is no longer in DooPlex's Longhorn backup**, D40 / R-1243) | operator (`kubectl`) | | 6 h (ep0, 14 daily + 8 weekly) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
|
||||
| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** **`[BETA-DEFERRED]`** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) |
|
||||
| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** **`[BETA-DEFERRED]`** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D **`[FACT]` 2026-10-03:** the PBS (whole-guest) off-site copy now also exists OFF Hetzner — DooPlex pulls ep0's datastore nightly (decision 70). The restic tier is still Hetzner-only. |
|
||||
| 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B. **D5 narrows this further:** Tier-1/2 app recovery needs only the drive, so losing R now costs the offsite route and host identity, never local app recovery (§7.4) |
|
||||
| 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix |
|
||||
| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **DEFECT** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |
|
||||
| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **FIXED 2026-10-05 (R-104, v0.297.0; was DEFECT — corrected 2026-10-24)** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |
|
||||
| 16 | **The container registry is lost with DooPlex** (the released images and agent packages) | the release copy on ep0 (D33): the newest 3 release tags of `felhom-controller`, `felhom-hub` and `felhom-samba`, and the newest 5 agent packages (binary + config bundle), in a plain folder outside PBS (`/srv/felhom-release-copy`) | push the copy back into a registry with the same name — `runbooks/total-loss-of-dooplex.md` „Restore the release copy” (same digests, no rebuild; `scripts/release-copy/`) | **operator** (ep0 root) | | 24 h (nightly 01:40) | **PARTIAL** | `audits/day-2026-10-23/08-release-copy-D33.txt`: the copy was pulled back, every sha256 and digest matched, and a controller image from it ran on a scratch box. Not yet walked: the push into a real new registry and the agent-package upload. Added 2026-10-25 (D44 = all picks, R-1265) |
|
||||
|
||||
### 8.2 The ONE exception to row 5's derived-copy rule (R-403, controller v0.230.0)
|
||||
|
||||
@@ -1590,6 +1662,19 @@ or pattern. **Why:** the data-loss hunt of 2026-10-20 proved two nights that del
|
||||
(R-1175: one share group's reconcile removed the other groups' copies, and a dead drive's copy; R-1181: a missing or
|
||||
emptied folder took its copy within one night). Controller: `internal/backup/tier2_kept.go`.
|
||||
|
||||
**`[DESIGN]` 2026-10-24 — a rejected backup block never deletes a copy (R-1252; review round 2 HUNT-2 H1, H2, H9;
|
||||
committed in controller `518a135`, unreleased).** The night's reconcile (deleting dest dirs the legs no longer name)
|
||||
runs ONLY under a valid classification. An app whose block is rejected (one bad line rejects the whole block) or absent
|
||||
falls back to the legacy legs, and nothing is reconciled that night — every existing copy stays. A REJECTED block puts
|
||||
„Az alkalmazás mentési beállításai most nem olvashatók…" on the app's second-drive status and tells the operator once a
|
||||
day (the operator-only digest `backup_run_failures`, leg `tier2-settings-unreadable`). A kept copy (D25) is always in
|
||||
the keep set, also when its leg later leaves the declared set (H2); Remove with „remove backups" drops the kept-copy
|
||||
records of what it deleted (H9). **`[FACT]` 2026-10-24 night, 9202 (`audits/restore-holes-2026-10-24/P2-9202/`):**
|
||||
audiobookshelf with a household file in `media/audiobooks`; two control runs kept the copy; one bogus path in the
|
||||
block → „the backup block is REJECTED … every existing copy on the second drive is KEPT", the copy there, the status
|
||||
sentence set, the operator digest sent (9202 has no hub: logged as dropped); the review's run on 0.342.0 the same day
|
||||
deleted that copy (`review-round-2-2026-10-24/proofs/H2-bad-backup-block/`).
|
||||
|
||||
### 8.2b The second-drive copy runs in a memory-capped container that never outlives the copy [DESIGN + FACT, R-1214, 2026-10-22/23 nights; rides controller 0.340.0]
|
||||
|
||||
The copy (`rsync -a --delete`) runs in a container of the controller's own image, capped at 512 MB, so its page cache
|
||||
@@ -1599,12 +1684,98 @@ refuses a missing one — then the in-process copy runs). **The container must n
|
||||
fix, 2026-10-23 night): a fixed name and the label `felhom.tier2-copy=1`; on any error, the timeout included, it is
|
||||
removed by name; at controller start, before the scheduler and the web server, every labelled container is removed;
|
||||
and a restore is refused while any second-drive copy runs or a leftover may still run (the other direction of R-1180 —
|
||||
a `--delete` beside a restore removes what the restore brings back). **`[FACT]` 9202, rc.5
|
||||
a `--delete` beside a restore removes what the restore brings back). *(As built — noted 2026-10-24: the refusal is in the
|
||||
web layer's `restoreOpBlocked` (`internal/web/restore_wizard.go:226`); the install-time „use my kept data" load checks
|
||||
only `RestoreStatus`/`IsRunning` (`internal/api/kept_install.go:89-92`) — a register row of review round 2.)* **`[FACT]` 9202, rc.5
|
||||
(`audits/night-2026-10-23/P1-straycopy/`):** a controller restart in the middle of a 16 GB copy → the new controller
|
||||
removed the old container at start; a restore probe during the copy was refused; the redone copy equals the source; a
|
||||
15-second test limit → the container removed, the partial copy stopped growing. Controller:
|
||||
`internal/backup/tier2_copy_lifecycle.go`.
|
||||
|
||||
### 8.2d An earlier install's unit is set aside, never rewritten [DESIGN, operator ruling D47, 2026-10-25 08:38; `09` §3 decision 256; R-1269 — controller `02daebe`, unreleased, rides v0.343.0]
|
||||
|
||||
**[FACT] Measured before (controller 0.342.0, which is `e1613ae` for these files):** the catalog sweep 2026-10-24
|
||||
(three times) and 9202 2026-10-25 (`audits/restore-truth-2026-10-25/before/`): Tandoor, one account, „Mentés most",
|
||||
remove keeping data and backups, install again. Within five minutes the periodic refresh rewrote ONLY the unit's
|
||||
`compose/app.yaml` (checksum `cbfcb538…` → `e19d2610…`) — the new install's database password — beside the OLD install's
|
||||
data. Its one „same definition" test was the image pins (`recovery_unit.go:194`, the R-696 `frozen` rule), and a
|
||||
reinstall of the same version has the same pins. A restore then paired the new password with the old data directory:
|
||||
`password authentication failed` ×44 (app) / ×23 (postgres), the app stayed `starting`, the account was gone from
|
||||
view. The next night's dump legs would have overwritten the old data as well, and the stale-drive prune deleted a unit
|
||||
on another drive whatever install it belonged to.
|
||||
|
||||
**[DESIGN] The rule.** A recovery unit's definition and its data are written only together, by a backup of the same
|
||||
install. Every install has an identity (`install_id` in its `app.yaml`): minted at a fresh install and at a `.fab`
|
||||
import, carried by every load-then-save, taken FROM the unit by a restore (the app then holds that unit's data), and
|
||||
minted once for an app installed before D47. The unit records it in its own `app.yaml`. Every writer of a unit asks
|
||||
first — the capture and its refresh, the night database dump, the pre-backup and the update's pre-backup dumps, the
|
||||
volume dump, the second drive's unit copy, the stale-drive prune. A unit of ANOTHER install (different identities; for
|
||||
a unit written before D47, a carried secret that differs — a reinstall mints new ones) is **moved whole** (a rename on
|
||||
its own drive; never a copy, never an edit) to `<namespace>/backups/set-aside/<app>/<UTC stamp>/` and recorded as a
|
||||
kept copy (§8.2a's record: kind `unit`, reason `earlier_install`); the new install then gets its own unit from its
|
||||
first backup. A move that cannot be done **refuses the write** and leaves the old unit byte for byte (an ERROR line
|
||||
names the app and both identities). Nothing moves a unit while that app is being restored.
|
||||
|
||||
**What the household sees.** The restore page lists the set-aside unit under „Megőrzött másolatok" („…· egy korábbi
|
||||
telepítés mentése") with **„Visszaállítás"** — the one unit-restore body, with the unit's own database password — and
|
||||
„Eltávolítás". Only those two, and the app's remove with „delete its backups too", ever remove it (the §8.2a / R-1252 H9
|
||||
rule). After a restore from it, the app is that unit's install; the app's own unit of the install it replaced is then
|
||||
set aside at the next backup — never lost.
|
||||
|
||||
**Known limit, stated.** A box that reinstalled and refreshed BEFORE this release already holds a rewritten unit; its
|
||||
old password is gone and nothing can bring it back. A unit from before D47 of an app with no carried secret has no
|
||||
signal: it is treated as this install's (the old behaviour) and gets the identity at its next write.
|
||||
**Why a volume app's reinstall asks no kept-data question:** the question (§6.5, decision 36) is about the app's drive
|
||||
FOLDER (`${HDD_PATH}` binds under `appdata/`); a volume app keeps nothing there — „remove, keep my data" removes its named
|
||||
volumes and keeps only its backups — so the install starts empty, and the kept backup is now the set-aside unit on the
|
||||
restore page. (The missing offer at install is its own row, R-1278.)
|
||||
|
||||
### 8.2e A restore whose app does not start is never „success" [DESIGN, R-1270, 2026-10-25 — controller `b680db8` + fix A; D49 = A (operator 2026-10-25); released in controller 0.343.0]
|
||||
|
||||
**[FACT] Measured before:** the health check after a restore was a WARN on every path, so a Tandoor restore whose app
|
||||
stayed `starting` was recorded `ok: true`, „…visszaállítva — az alkalmazás újraindult." (catalog sweep 2026-10-24; 9202
|
||||
2026-10-25, `audits/restore-truth-2026-10-25/before/`). **[DESIGN]** Every restore path — the unit restore (so the
|
||||
second drive's unit and whole restore, a kept load, a set-aside unit), the off-site restore, the second drive's file
|
||||
restore and the no-unit fallback — returns „not started" when the app does not reach running within the health wait
|
||||
(90 s). The household reads „A(z) %s: az adatokat visszatöltöttük, de az alkalmazás még nem indult el. Várj pár percet,
|
||||
majd nyisd meg újra. Ha így marad, szólj nekünk." (the record `ok: false`, `not_started: true`); the operator gets one
|
||||
`backup_run_failures` line (leg `restore-not-started`). **The safest reading, the operator may change it:** a timeout
|
||||
does NOT pour the old data back — a slow first start (a database migration) is common, and a put-back would throw away a
|
||||
correct restore. The data stays restored and the restore's undo copy is KEPT — the same „kept for support" state a failed
|
||||
put-back leaves (§8 D20/D22), so no new retention rule. A watcher looks 30 more minutes: when the app runs, the record
|
||||
turns to success („…elindult: a visszaállítás kész.") and the copy is removed, as after any successful restore; if it
|
||||
never runs, the copy stays for support.
|
||||
|
||||
**D49 = A (operator ruling 2026-10-25, attended release run; `09` §3 decision 258):** the safest reading above is the rule —
|
||||
the box keeps the restored data, keeps the old copy, tells the operator, and never puts the old data back by itself.
|
||||
**Fix A (controller 0.343.0, R-1270 follow-up):** „started" is one helper on every path — the stack running on every
|
||||
sample of a 45 s window, no container's `RestartCount` grown in it, every container with a healthcheck `healthy` (a
|
||||
container without one is judged by running alone). An app that starts and then crash-loops is „not started". The undo
|
||||
copy's RELEASE is a separate, longer test: 3 minutes of the same after the verdict (10 s samples, watched up to 30
|
||||
minutes). **[FACT]** 9202 rc.6: an image that exits after 20 s → not started, copy kept, one operator line; the original
|
||||
unit back → success, copy released 3 minutes later. Tester 1 on 0.343.0: a unit restore and a set-aside restore, each
|
||||
„ran stably for 3m0s … the restore copy is released" (`audits/release-2026-10-26/`).
|
||||
|
||||
### 8.2c Decisions in code that no document recorded [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
|
||||
|
||||
Read at `file:line` on `main` (controller `321df1f`, hub `44d19042`) by the main session:
|
||||
|
||||
- **A second-drive copy is limited to 60 minutes**; a stray copy container that cannot be removed blocks every
|
||||
restore and is retried every minute; copies into one destination run one at a time and are never killed by name
|
||||
while running (`felhom-controller/controller/internal/backup/tier2_copy_lifecycle.go:40,113-121,150-172`).
|
||||
- **The household's „Eltávolítás" of a kept copy (D25) is refused while any backup, restore or second-drive copy runs**
|
||||
(`internal/web/kept_copies.go:54-58`).
|
||||
- **A start that fails AFTER a successful put-back** leaves the app stopped and NOT held, keeps the copy, and tells the
|
||||
household `restore_failed_put_back_not_started` (`internal/backup/offbox_restore_undo.go:245-247`).
|
||||
- **The hub's off-site delivery checker** (`hub/internal/monitor/offsite_delivery.go`): a box that took no off-site
|
||||
credential is reported `offsite_delivery_stuck` (operator) and healed by `maybeHeal` (a burned vs a stuck shape);
|
||||
a box that declares it waits for the household's recovery code (`awaiting_recovery_key`) is skipped — no event, no
|
||||
heal (R-1230, `:103-110`).
|
||||
- **The release copy (D33)** — nightly 01:40, the newest 3 image tags and 5 agent packages to ep0 under its own
|
||||
limited user, capped by the sender (`felhom.eu/scripts/release-copy/`). ~~has no row in this matrix~~ It is row
|
||||
**16** of the matrix since 2026-10-25 (D44 = all picks, R-1265); its other homes are the poster facts and
|
||||
`runbooks/total-loss-of-dooplex.md`.
|
||||
|
||||
### 8.1 The blank cells, listed explicitly
|
||||
|
||||
Per the rule that a blank is a finding, here they are:
|
||||
@@ -1622,6 +1793,7 @@ Per the rule that a blank is a finding, here they are:
|
||||
| 13 | RTO, RPO | not a timed recovery; a capability loss |
|
||||
| 14 | RTO | the break-glass path is proven but was never timed |
|
||||
| 15 | RTO | the manual unlock was performed but not timed |
|
||||
| 16 | RTO | a registry rebuild from the ep0 release copy has never been timed (row added 2026-10-25, D44 #6, R-1265) |
|
||||
|
||||
**Also unmeasured, and not representable as a row** (INV Part F.3): any restore larger than 155.5 MB
|
||||
from restic; any restore of a customer data drive (no tier holds one whole); the weekly PBS
|
||||
@@ -1875,7 +2047,7 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one.
|
||||
| host-loss plan → an actual restore | **EXECUTES NOTHING BY CONSTRUCTION** | `felhom-agent/internal/dr/plan.go:1-4` |
|
||||
| host rebuilt as its former self | **NEVER DONE** | INV Part D1 |
|
||||
| escrow **consume** in a real recovery | **SPIKE-LEVEL ONLY** | `06-offsite-connectivity.md:327` |
|
||||
| hub DB restore from its Longhorn backup | **NEVER DONE** | INV Part D2.2 |
|
||||
| hub DB restore from its ep0 copy into a running hub (the weekly restore-test opens the copy and checks it; a hub has never been brought up on it) | **NEVER DONE** | INV Part D2.2; the Longhorn backup it used to cite is off since 2026-10-24 (D40) |
|
||||
| a **customer** performing a restore unassisted | **MISSING AS EVIDENCE** | `00-capability-map.md:75` |
|
||||
|
||||
---
|
||||
|
||||
@@ -135,7 +135,9 @@ live 2026-08-23 — one event across 22 scans.
|
||||
|
||||
**The vocabulary is the HUB's and it is exact: `{info, warning, error, critical}`.** Anything else is
|
||||
**coerced to `info` at ingest**, and `info` is dropped by `severityNotifies` before *both* delivery
|
||||
legs. So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
|
||||
legs — except the paired recoveries in `recoveredPairedDownTypes` (`node_recovered`, `host_recovered`,
|
||||
`whole_guest_backup_recovered`, `pbsdr_box_recovered`, `offsite_box_recovered`, `storage_reconnected`,
|
||||
`health_recovered`), which run before the gate and are mailed (`notify/dispatcher.go` — corrected 2026-10-24, review round 2). So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
|
||||
and is e-mailed to **nobody**.
|
||||
|
||||
**This shipped twice.** `DiskAlertKind.Severity` emitted `"warn"` until controller v0.215.0;
|
||||
@@ -155,8 +157,10 @@ Three things now hold it:
|
||||
3. **The dispatcher's `unrecognized severity` branch is kept**, because the hub's own monitor checkers
|
||||
call `ProcessEvent` directly and never pass the ingest handler. For them it is the only guard.
|
||||
|
||||
**Who gets it.** `processOperator` consults only `operatorOn`, the address and a 1-hour cooldown —
|
||||
**never customer preferences** — so a valid severity always reaches the operator. `processCustomer`
|
||||
**Who gets it.** `processOperator` consults only `operatorOn`, the address and a cooldown —
|
||||
1 hour, except 5 min for node/host liveness and 6 h for a storage-fill warning (`operatorCooldownFor`); it records a
|
||||
`backup_tier_skipped` in a box's FIRST hour as „suppressed" instead of mailing it (R-723), and the mail hold applies —
|
||||
**never customer preferences** (cooldowns and R-723 corrected 2026-10-24) — so a valid severity always reaches the operator. `processCustomer`
|
||||
consults `operatorOnlyEvents` and then the customer's `enabled_events`.
|
||||
|
||||
**`app_start_failed` is customer-switchable but OFF by default** [DESIGN, operator ruling 2026-08-23]:
|
||||
@@ -180,7 +184,10 @@ rather than assumed (2026-08-30):
|
||||
- **not** in `operatorOnlyEvents` — the operator leg ignores customer preferences anyway, so the
|
||||
operator is always mailed; putting it here would only remove the customer's ability to opt in.
|
||||
- **not** in `DefaultEnabledEvents` — customer-switchable, default OFF, the same ruling as
|
||||
`app_start_failed`. The checkbox already exists at `settings_notifications.html:34`.
|
||||
`app_start_failed`. **Superseded by operator Q1 (2026-10-12, R-388 slice 1):** since controller v0.309.0
|
||||
`backup_integrity_failed` (with `crossdrive_failed`, `storage_reconnected`, `health_recovered`) IS in
|
||||
`DefaultEnabledEvents` — the default for a household with no saved list (`internal/settings/settings.go`, pinned by
|
||||
`TestR388_DefaultListAsGroups`; dated note 2026-10-24, review round 2). The checkbox already exists at `settings_notifications.html:34`.
|
||||
|
||||
**And a caveat that belongs in the alarm ladder rather than only in the backup document:** a
|
||||
`backup_integrity_ok` at the shipped depth means *the index, the pack inventory and the snapshot graph
|
||||
@@ -293,6 +300,7 @@ bands.
|
||||
| app down (`app_start_failed`) | **per APP** | `…:<stack_name>` | no digest exists for it — see below |
|
||||
| update outcome (`app_update_undone`, `app_update_held`) | **per APP**, both legs | `…:<stack_name>` | hub v0.120.0 — one mail per app per outcome; the household leg has its own register (`perAppCustomerCooldownEvents`) |
|
||||
| OOM storm (`app_oom_storm`) | **per APP**, operator | `…:<stack_name>` | hub v0.121.0 — no digest; the controller already sends it at most once per container run |
|
||||
| more per-APP types (added 2026-10-24 from `dispatcher.go`'s per-stack set) | **per APP** | `…:<stack_name>` | `app_hold_no_whole_copy`, `app_stopped_unhealthy` (both legs), `app_repaired_after_crash`, `app_unexplained_upload`, `app_unexplained_cpu` |
|
||||
| backup run (`backup_run_failures`) | per RUN | `…:<run_id>` | a digest already lists every failing app; one per run |
|
||||
| tiered backup (`whole_guest_backup_failed`, …) | per TIER | `…:<tier>` | the tiers fail independently and mean different things |
|
||||
| everything else, incl. `crossdrive_failed` and `backup_integrity_failed` | per TYPE, per hour | — | coarse **on purpose** |
|
||||
@@ -367,6 +375,14 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
|
||||
|
||||
---
|
||||
|
||||
**More operator-only box alarms, listed 2026-10-24 (review round 2; each is in `notify/dispatcher.go`'s
|
||||
`operatorOnlyEvents`, and each has its home elsewhere):** `wg_tunnel_stale` / `wg_tunnel_ok` — the box's tunnel to ep0:
|
||||
a handshake older than 6 h or stamped in the future, edge-triggered (D30, R-1204; `api/host_tunnel.go`, `05` §15.4b);
|
||||
`host_swap_short` / `host_swap_ok` — the host's swap below 90 % of min(the guests' swap allowance, 8192 MB) (R-1192;
|
||||
`api/host_swap.go`); `bind_old_host_linked` — a household asked for a new bind link while its old host is still linked
|
||||
(R-1203; `05` §13); `golden_not_vouched` — a published golden the hub does not vouch (decision 199, R-242);
|
||||
`recovery_older_package` — a household's code opens, or may open, an older sealed package (decision 183, R-304 C).
|
||||
|
||||
## 6.4 A box that is not always on: the missed-backup deadline and the household's outage mail [DESIGN, hub v0.134.0, 2026-10-05]
|
||||
|
||||
Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *operator may reverse*).
|
||||
@@ -376,7 +392,8 @@ Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *op
|
||||
never judged (measured 2026-10-05: `1 skipped (down)`). Now a down box is judged on longer lines:
|
||||
`expected_dbdump_missed` after **48 h** without a `db_dump_completed`, `expected_backup_missed` after **72 h**
|
||||
without a whole-guest backup in any retained host report — never for a box first seen less than 48 h ago. A box that
|
||||
died last night still raises only its staleness alarm. A `disabled` box is still skipped (R-321). Pinned by
|
||||
died last night still raises only its staleness alarm. A `disabled` box is still skipped (R-321), and so is a box the household switched off on purpose — by its staleness
|
||||
state or by the host row's hold (R-998, `monitor/deadline.go`; corrected 2026-10-24). Pinned by
|
||||
`monitor/r872_down_box_test.go`.
|
||||
- **R-873 — "your server cannot be reached" reaches the HOUSEHOLD at most once per 7 days** (`node_stale`,
|
||||
`node_down`, `host_stale`, `host_down`, read from the persisted notification log, so a hub restart does not reset
|
||||
@@ -387,7 +404,7 @@ Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *op
|
||||
|
||||
---
|
||||
|
||||
## 6.5 One lost off-site copy on a pinned tier [DESIGN, R-435 — `09` §3 decision 191, hub main 2026-10-08, unreleased]
|
||||
## 6.5 One lost off-site copy on a pinned tier [DESIGN, R-435 — `09` §3 decision 191; released in hub v0.144.0 — corrected 2026-10-24]
|
||||
|
||||
On a **pinned** off-site tier (the hub holds a confirmed append-only key, decision 69) the only legitimate way the
|
||||
snapshot count can fall is a clean-up window the hub opened (decision 68). So the hub raises
|
||||
@@ -470,7 +487,7 @@ was ruled that way on its own merits and does not pre-judge the redesign.
|
||||
**Filed as a PRODUCT DECISION, not a defect** — see the register. It is the operator's call to take
|
||||
separately, and no part of it was implemented here.
|
||||
|
||||
### 8.1 The design — „only what you can act on" [DESIGN, R-388 — `09` §3 decision 200, 2026-10-12; NOT BUILT]
|
||||
### 8.1 The design — „only what you can act on" [DESIGN, R-388 — `09` §3 decision 200, 2026-10-12; slice 1 BUILT in controller v0.309.0 — three group switches, details under Részletek (`internal/settings/notif_groups.go`); corrected 2026-10-24]
|
||||
|
||||
*Operator ruling 2026-10-12 (S5): design now, build in slices later. Written by CC from the direction above and the
|
||||
live page (controller, `settings_notifications.html`: **18 event switches** plus the address, the cooldown and the
|
||||
@@ -531,3 +548,53 @@ list is shown as custom and saved back unchanged; the default list maps to group
|
||||
multi-day backup fault reads as „nobody noticed" if the household ever finds out.
|
||||
2. **The cooldown field („hány óránként legfeljebb"):** keep it on the page or move it under „Részletek"? Pick:
|
||||
**under Részletek** — it is an engineering choice, which is what this redesign removes.
|
||||
|
||||
## 9. The upload and CPU watch — „does the app explain its own traffic?" [DESIGN — built; released in controller v0.342.0 and hub v0.169.0 (deployed) on the operator's yes, 2026-10-24 — corrected 2026-10-24, review round 2]
|
||||
|
||||
**Why** (operator ruling 2026-10-23 19:22, `CONTEXT.md`): DooPlex's hacked Plex sent ~474 GB in a day with no viewer and
|
||||
used ~10 cores with no film being converted (`audits/plex2shell-felhom-impact-2026-10-23/`). Nothing on a Felhom box
|
||||
would have noticed: the health check sees whole-box CPU at one instant, and nothing looks at upload at all. **The idea:**
|
||||
when an app sends a lot (or burns CPU) for a long time, ask the APP whether that is its work. Work the app explains is
|
||||
normal use; work it does not explain is the sign of a hacked app. Numbers and measurements:
|
||||
`audits/SPIKE-upload-watch-2026-10-23.md`; the code: controller `internal/upwatch` (README §8e).
|
||||
|
||||
1. **The line.** Learnt from what the box already does: while its off-site push runs, the controller samples its own
|
||||
sent bytes once a second; the best ≥ 10-second rate is kept per night (14 nights, settings.json `upwatch.line`); a
|
||||
night counts when that window moved ≥ 10 MB; the line is the highest counting night. A **lower bound**, shown so
|
||||
(„legalább N Mbit/s", system settings). **AMENDED 2026-10-24 (operator ruling 07:26, R-1247): the weekly line TEST.**
|
||||
The night's one-second read showed how low that floor is (demo-hp „at least 10 Mbit/s"; spike §2.1). So the host's
|
||||
agent (agent ≥ 0.164.0, `internal/linetest`) runs a 10-second iperf3 upload once a week to ep0's tunnel-only server:
|
||||
Tuesday 10:00–15:59 local from a per-host slot, only after 10 quiet minutes of the physical uplink (< 2 Mbit/s each
|
||||
minute), under the one-heavy-op gate, best of 3, never in the night chain; „server busy" → later the same day. The
|
||||
controller reads it (`GET /host/line-test`) every 15 minutes; the watch judges against the higher of test and floor;
|
||||
the system page says „N Mbit/s (heti sebességméréssel)". demo-hp measured 90.7 Mbit/s (`audits/line-test-2026-10-24/`).
|
||||
2. **Trigger, per app** (its compose project's containers summed; exact counters from the Docker Engine API, once a
|
||||
minute): upload ≥ max(5 Mbit/s, 25 % of the line) for 15 minutes; CPU ≥ max(150 %, 50 % × cores) of one core for 30
|
||||
minutes. Infrastructure (the controller — its own push is the line measure —, traefik, cloudflared, Samba) is never
|
||||
judged; the box's own work on an app (a backup run, an update, a deploy) is never judged.
|
||||
3. **The explanation.** qBittorrent (`up_info_data`) and Transmission (`uploadedBytes`): explained when the container
|
||||
sent ≤ 1.25 × what the app counted + 1 Mbit/s. Plex: any viewer explains upload; any transcode explains CPU. An app
|
||||
that does not answer gives no number (a hacked app may refuse its own API). No explainer: upload alarms only at ≥ 10 ×
|
||||
its own p95 for 60 minutes after ~20 h of history; CPU at ≥ max(200 %, 75 % × cores) for 60 minutes.
|
||||
4. **Who is told** (the R-388 rule, §8.1): **`app_unexplained_upload` / `app_unexplained_cpu`** (warning) and their
|
||||
`_cleared` (info, recorded) are **operator-only** (hub `operatorOnlyEvents`), one per episode, per-app cooldown. The
|
||||
household is told only about the qBittorrent limit, because they can change it (R-1246 asks the operator).
|
||||
5. **qBittorrent's default upload limit:** 50 % of the TESTED line (D41 = 50 %, operator 2026-10-24; never of the
|
||||
push floor), every 15 minutes, re-set when the line moves > 20 %; never over a limit the household set (one there
|
||||
first, or any change from the value qBittorrent stored for the box — it keeps KiB/s). No test → none set. **A box
|
||||
that seeds all day (R-1250, brief 2026-10-24; committed agent `29bcf44`+`1e087bb`, controller `046b808`, not yet
|
||||
released):** the 10 quiet minutes never come, so when the agent reports `waiting_for_quiet` (due, in the window,
|
||||
tunnel up, the full 10-minute window watched and busy) and qBittorrent or Transmission runs, the controller writes
|
||||
`upwatch.line_test_hold` (the app's limit exactly as stored) FIRST, holds the app at 1 KiB/s (Transmission 1 kB/s),
|
||||
calls `POST /host/line-test/run` (5 s settle, 10 s quiet check < 2 Mbit/s, three runs 5 s apart — about 55 s), then
|
||||
puts the old limit back, the household's own included; a change made during the test stays; a killed controller's
|
||||
start-up check puts it back; the 50 % rule waits while a record exists. Measured on demo-hp with a real seed (10
|
||||
Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; with it the uplink fell to 0.02–0.16 Mbit/s,
|
||||
94.2 Mbit/s measured, the app held 56.8 s and back to 1 249 280 B/s exactly (`audits/dooplex-alarms-2026-10-24/`).
|
||||
While the box owns it, qBittorrent's page says: „A qBittorrent
|
||||
feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
|
||||
6. **Limits, stated:** *(added 2026-10-24, review round 2: ANY viewer explains any amount of upload, so a hacked app can send freely while one real playback runs — `upwatch/watch.go:275,299`; R-1262)*; an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
|
||||
a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood
|
||||
whatever the API says; not built. The box cannot tell LAN from internet bytes per app; a host-network container
|
||||
(Samba) has no counter; the push floor is a lower bound, the weekly test is close to the line.
|
||||
7. **Release:** released 2026-10-24 on the operator's yes (controller v0.342.0, hub v0.169.0; R-1244) — corrected 2026-10-24.
|
||||
@@ -65,6 +65,13 @@ build is added to both in the same commit.
|
||||
|
||||
### 1.1 The button
|
||||
|
||||
> *Dated note 2026-10-24 (review round 2): §1 is the 2026-09 MEASUREMENT before the arc. `UpdateStack` no longer exists
|
||||
> (deleted in v0.237.0, controller `REUSE.md`); the button calls `StartGuardedUpdate` — backup precondition, safety
|
||||
> dump, pin journal, undo copy, health wait, undo (`internal/api/router.go:794`; `internal/stacks/update.go`). The line
|
||||
> numbers below are the measurement's; e.g. `copyTemplates` is now `sync.go:436`, `copyIfChanged` `:700`, the `15m`
|
||||
> default `config.go:399`, `RestartStack` `manager.go:1454` (it now runs `up -d --force-recreate` and errors when no
|
||||
> container restarted, R-1026).*
|
||||
|
||||
`Manager.UpdateStack` (`felhom-controller/controller/internal/stacks/manager.go:1199`) is two compose
|
||||
commands and nothing else:
|
||||
|
||||
@@ -267,7 +274,8 @@ builds them (`audits/update-rulings-2026-09-23/`).
|
||||
next night. **Why:** the update then leans on the freshest copy the box ever has, and there is no
|
||||
second window for anyone to set or misread. **Replaces:** §3b Q1's recommendation of a fixed
|
||||
02:30–05:00. *For the builder:* `stacks.update_window` (`config.go` L195, default `03:00-05:00`)
|
||||
exists and **no Go code reads it** — it appears only in `config.go` (field + default),
|
||||
exists and **no Go code reads it** *(dated note 2026-10-24: the field is now removed — no controller source names
|
||||
it; `controller/README.md` „`stacks.update_window` is removed")* — it appears only in `config.go` (field + default),
|
||||
`setup/handlers.go` L482 (written into a new box's `controller.yaml`) and
|
||||
`configs/controller.yaml.example`. Remove it or fold it into the chain; do not build a second
|
||||
window.
|
||||
@@ -278,17 +286,27 @@ builds them (`audits/update-rulings-2026-09-23/`).
|
||||
13. **The test decides, not the tag** (Q3). **REPLACES decision 3's second half, "never across a
|
||||
major".** The box applies by itself every step the catalog holds, because **the catalog holds only
|
||||
tested steps**; the size of the version jump does not matter, the test does. The box does not
|
||||
parse tags to decide. The tag rule (`stacks.CompareImageRefs`) moves to the **catalog gate** as a
|
||||
push-time safety net: an image move with no test record is refused there. Two exceptions, both
|
||||
parse tags to decide. ~~The tag rule (`stacks.CompareImageRefs`) moves to the **catalog gate** as a
|
||||
push-time safety net:~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The tag rule (`stacks.CompareImageRefs`) did not move: it
|
||||
stays in the controller (`internal/stacks/updateorder.go`), where it only tells whether an installed app is
|
||||
AHEAD of the catalog; it never decides an update. The catalog gate is the push-time safety net, and it asks for a
|
||||
proven test instead (§6.4 part 4): an image move with no test record is refused there. Two exceptions, both
|
||||
**marks the test sets on the step**: *files may change* → automatic only when a fresh copy holds
|
||||
the files, else a person (this is Q2's refinement); *needs a person* → the tester writes why.
|
||||
**Why:** decision 3 was a proxy. "Within a major" was standing in for "known to work", and the
|
||||
update night measured the proxy wrong in both directions — minor moves that held
|
||||
(adventurelog, outline) and a tag shape it cannot read at all (§3b Q3). The test record is the
|
||||
thing the proxy was guessing at.
|
||||
**Dated note 2026-10-23 (decision 255, D39 = B):** a second step kind now exists — the `security_fix` step, proven
|
||||
on the box with the bench optional, for an urgent upstream fix with a public notice. Decision 13's rule is
|
||||
unchanged for every other move: a step a box takes was still tested, on at least the box venue.
|
||||
|
||||
14. **The update ladder** (Q3). A box more than one step behind **climbs one tested step at a time, in
|
||||
order, and never jumps**. Each step is the full guarded update of §6.1. **Why:** a step was tested
|
||||
order, and ~~never jumps~~ the night leg never jumps**. Each step is the full guarded update of §6.1.
|
||||
**[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** An app older than the ladder (its installed version matches no entry's `from`) is never
|
||||
pressed by the night leg (`internal/stacks/unattended.go`). A PERSON's press on such an app takes the catalog's
|
||||
current definition, logged by name (`internal/stacks/ladder.go`, `TestLadder_UnknownInstalledJumpsAndSaysSo`); the
|
||||
guarded update and its undo (decision 15) still run around it. **Why:** a step was tested
|
||||
from the version before it, not from three versions before it; R-40's multi-major jump is what
|
||||
this prevents. This is the "stepping" §6.1 already assigned to slice 6.
|
||||
|
||||
@@ -421,7 +439,8 @@ R-636's louder repeated alarm.
|
||||
(a) broke the second. Shipped as controller **v0.269.1** — a second release in the session, instead of
|
||||
shipping (a) to the fleet with the floor.
|
||||
*Note 2026-09-30 (R-740) — the cost line above is not what the code does.* „(or the automatic leg)": the
|
||||
automatic leg never presses a digest-only change (`unattended.go:433–435`, skip `no_test_record`). And the
|
||||
automatic leg never presses a digest-only change (`unattended.go:433–435` then, `:449-451` now — line corrected
|
||||
2026-10-24 — skip `no_test_record`). And the
|
||||
catalog never writes a re-test of a floating tag at a new digest (the writer refuses a step that moves no
|
||||
image), so a same-name upstream fix reaches no box by either route today. Measured, with the options, in
|
||||
R-740; the decision is the operator's (STATUS 2026-09-30). This decision's ruling — the sync never moves
|
||||
@@ -976,8 +995,11 @@ its length, and both fixes cost something the household would notice — operato
|
||||
installs the vouched golden and updates itself to the floor (not yet proven end to end — R-1052). **The golden
|
||||
stays weekly:** a CHANGED golden selection is still refused when it is older than the fleet's newest controller
|
||||
(`TestVouchGoldenChangedToOlderIsStillRefused`).
|
||||
5. The push gate judges the golden by its AGE (older than 7 days while the floor is above it), not by the newest
|
||||
CHANGELOG heading (`scripts/golden_currency_gate.py`).
|
||||
5. The push gate judges the golden by its AGE, not by the newest CHANGELOG heading
|
||||
(`scripts/golden_currency_gate.py`). **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** ~~(older than 7 days while the floor is above
|
||||
it)~~ The gate runs offline and cannot read the floor, so „a release is ahead of the newest bake” stands in for
|
||||
„the floor is above it”: it convicts when a release is ahead AND the newest bake is more than 7 days old; ahead
|
||||
and at most 7 days old is an advisory. The floor follows a release within hours, so the gate is the stricter one.
|
||||
251. **Scripts use scoped hub tokens, not the operator password; CC's floor step uses its own.** *Operator ruling
|
||||
2026-10-15 11:38.* Tokens with groups (read/write per group), managed on the Configuration tab, shown once, stored
|
||||
hashed, every use logged by name (`05` §17, hub v0.152.0). Never by any token: a secret shown or minted, a delete or
|
||||
@@ -1031,6 +1053,87 @@ its length, and both fixes cost something the household would notice — operato
|
||||
Built in controller v0.314.0 (`internal/stacks/restore_gate.go`); pinned by `restore_gate_test.go` and
|
||||
`r1057_unit_gate_test.go` (ten mutations each seen failing). Proof: `audits/gate-restore-2026-10-15/9202/`.
|
||||
|
||||
### 2026-10-23 (~12:00) — D39 = B: the security-fix step (operator ruling by starting the run; R-1239)
|
||||
|
||||
255. **An urgent upstream security fix reaches the boxes at night as a `security_fix` ladder step: the box walk is
|
||||
required, the bench is optional, and a public notice is the authority.** *Operator ruling 2026-10-23 ~12:00 (D39 =
|
||||
B, R-1239), taken by starting the D39 brief; its design is Claude.ai's recommendation, taken with it.* Why: on
|
||||
2026-10-23 the only route for the Plex2Shell fix was an operator word, `git push --no-verify` past the move gate,
|
||||
and a hand press on each box — and the night leg never takes a step with no test record (R-1238). That does not
|
||||
scale past the first customers and needs the operator awake.
|
||||
1. **The entry** (`scripts/ladder.py` `check_security`, the one reader): `"verdict": "security_fix"` with `security`
|
||||
`{ids: [CVE-…|GHSA-…], advisory: "https://…", fixed_in}`, `box_evidence` (required), `seed` `read_back`|`none`,
|
||||
`memory_peak_pct` from the BOX soak with `memory_basis: "box_soak"`; `evidence` = the bench record or null — a
|
||||
bench given must be `proven` (`bench_verdict`, `harness_version` ≥ 2 ride with it). Never `backfilled`,
|
||||
`digest_from` (a same-tag fix stays decision 52's lane), `engine_conversion`, nor a database engine MAJOR (any
|
||||
engine — MariaDB included; the judgement is `check-engine-major.py`'s own `engine_of`/`major_of`, imported).
|
||||
2. **No operator word per incident.** The session that writes the step reports it to the operator after the push.
|
||||
3. **The gates:** the move gate accepts a `security_fix` move exactly as a proven one (from/to per service, digests
|
||||
the registry still serves); `check-test-record.py` WARNs on every push while a template's head is a security-fix
|
||||
step with no bench test („<app>: security-fix step <to> has no bench test (since <tested_at>)"). Seven decoys
|
||||
(`test_gate_decoys.py`), each seen failing with the rules switched off.
|
||||
4. **The box venue:** `box_walk.py --security` on 9202 through the product's Update; verdict `proven` only if the
|
||||
update ended `done` with no undo and no hold, the app runs, the four version observables are TO, the seed (if
|
||||
any) reads back, and a soak (default 600 s, cgroup `anon` against the limit, OOM kills and restarts counted)
|
||||
sees no kill and no restart. A negative control (a TO image that exits at once) comes back `failed`.
|
||||
5. **The writer:** `upgrade-test.py --write-ladder --security …`, refusing a box walk not proven, a bench given and
|
||||
not proven, a template not at the walk's FROM, a digest that moved since the walk, an engine major, a missing
|
||||
field. The compose move, `catalog_since` and the superseded step's own files are the shared `apply_entry`.
|
||||
6. **The controller (v0.341.0):** the leg takes `proven` OR `security_fix`; every other guard runs unchanged, in
|
||||
the same order; it logs `[update-leg] <app>: security fix <ids> taken (<from> → <to>)` and the hub report's
|
||||
step carries the ids. A person's press and the badge are unchanged. **A controller ≤ 0.340.0 skips the step at
|
||||
night (`not_proven`) and a hand press takes it** — proven on 9202, below.
|
||||
7. **The trade, stated.** A security-fix step skipped the bench. For an app with a box fixture the data is still
|
||||
read back after the update; for an app without one (Plex today) the step is checked only by „it starts, it is
|
||||
healthy, it runs the new version, and a 10-minute soak sees no kill or restart". `files_may_change` is a bench
|
||||
measurement: with no bench the step carries `false`. That is accepted because the leg's own guard is not the only
|
||||
net — the update takes a per-app backup first, the undo puts the pre-update data back on a failed health check
|
||||
(decision 15), and a security fix that rewrites files without breaking the health check is the case the bench
|
||||
would catch and this lane accepts the risk of. A session that knows a fix migrates files runs the bench.
|
||||
**Proof (9202, drill catalog, 2026-10-23):** `audits/security-lane-2026-10-23/` — the negative control (TO
|
||||
`busybox:1.36.1`) came back `failed` (the box undid it after 335 s) and the writer refused it; Plex 1.43.4.10903 →
|
||||
1.43.5.11029 `proven` (done in 36.9 s, the four observables at TO, digest `sha256:01bdf0ad…5c5f`, soak 600 s: anon
|
||||
peak 10.7 % of 2 GiB, 0 kills, 0 restarts, `seed: none`); the writer wrote the drill step (CVE-2026-96652, -96654,
|
||||
-96655, -96656; Plex's own notice as CISA KEV cites it, added 2026-10-20; fixed in 1.43.3.10861); on 0.340.0 the
|
||||
night leg logged `plex: skipped — not_proven` and a hand press took the step; on 0.341.0-rc.1 the night leg logged
|
||||
`plex: security fix CVE-2026-96652,CVE-2026-96654,CVE-2026-96655,CVE-2026-96656 taken` and ended `done` in 30.1 s
|
||||
after the update's second-drive copy and undo copy. Released as controller 0.341.0, global floor 11:24 UTC.
|
||||
|
||||
### 2026-10-25 (attended release run) — D49: a restore that does not start keeps both copies; D44–D46: the review picks
|
||||
|
||||
258. **A restore whose app does not start keeps the restored data AND the old copy, tells the operator, and never puts the
|
||||
old data back by itself (D49 = A).** *Operator ruling 2026-10-25, given by starting the attended release run.* Options:
|
||||
A) keep and wait — cost: an app that never starts needs a person; B) pour the old data back after the wait — cost: a
|
||||
slow but correct first start (a database migration) is thrown away. Why A: the slow first start is the common case,
|
||||
the old copy is kept, a late start turns the record into a success. Built: `07` §8.2e and fix A (the verdict needs a
|
||||
stable start; the copy goes only after 3 more minutes of it) — controller 0.343.0.
|
||||
259. **The 21 confirm lines and four code changes of the second review round are accepted as picked (D44 = all picks).**
|
||||
*Operator, 2026-10-25.* The notes carry a normal `[DESIGN — confirmed …]`; the code changes became R-1280–R-1283
|
||||
(R-1265 closed).
|
||||
260. **Where a design document and the code disagree on the seven named points, the text follows the code (D45 = all
|
||||
picks).** *Operator, 2026-10-25.* Each corrected place is tagged `[DESIGN — corrected 2026-10-25, D45 = all picks,
|
||||
R-1266]`, the old words struck through, not deleted (R-1266 closed).
|
||||
261. **Homepage is behind the family gate, like Glance (D46 = yes).** *Operator, 2026-10-25.* Why: it has no login of its
|
||||
own and showed a stranger the box's CPU, memory and disk. Tried on the drill first; catalog `ebbf1f4` (R-1272 closed).
|
||||
|
||||
### 2026-10-25 (08:38) — D47: nothing changes an existing backup; D48: a restore downloads first (operator rulings by starting the run; R-1269)
|
||||
|
||||
256. **An existing recovery unit is never changed in part (D47).** A unit's definition (compose, pins, `app.yaml`) and
|
||||
its data are written only together, by a backup of the same install. A new install never adopts, rewrites or
|
||||
deletes a unit that an earlier install left; that unit is set aside whole and stays restorable. *Operator ruling
|
||||
2026-10-25 08:38, taken by starting the R-1269 brief.* Why: on 2026-10-24 the catalog sweep measured three times
|
||||
that a reinstall's own five-minute refresh wrote the new install's database password into the unit „remove — keep
|
||||
backups" had kept; a restore then could not open the old data (R-1269). Built: every install has an identity
|
||||
(`install_id`), the unit records it, and every writer asks first — `07` §8.2a „An earlier install's unit is set
|
||||
aside" (controller, unreleased; rides v0.343.0).
|
||||
257. **Every restore path resolves and pulls every image its definition names before the first change to the app; a
|
||||
failed pull is a refusal with the app untouched (D48).** *Operator ruling 2026-10-25 08:38, taken by starting the
|
||||
brief.* Why: the restore stopped the app and moved its data first and pulled only inside the database-only start;
|
||||
a version that could not be downloaded then ended in the D22 put-back, a full stop-and-pour-back for nothing.
|
||||
Built: `07` §6.6 „The limit — the image is not in the backup" (controller, unreleased; rides v0.343.0).
|
||||
**Confirmed, no change:** `07` §6.6 — a restore brings the app back at its data's version, then the guarded update
|
||||
climbs the ladder; R-698 (a deleted version cannot come back) and R-933 (our own image mirror) are unchanged.
|
||||
|
||||
### 2026-10-15 (07:48–08:20) — bake first; Plex through its own key; HSTS one day (recorded before the media build)
|
||||
|
||||
247. ~~**Bake first.** A controller version is RELEASED only when its golden is baked.~~ **REVERSED by decision 250,
|
||||
@@ -1932,7 +2035,7 @@ moves a stack dir.
|
||||
| writer | pin source |
|
||||
|---|---|
|
||||
| the deploy path (`runComposeDeploy`) | the template just deployed from |
|
||||
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull |
|
||||
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull — *as built: `UpdateStack` was deleted in v0.237.0; the guarded update writes the pin (`advancePinTo`, the `pinning` phase before `pulling`), and two more paths move a pin BACK — `pinBack` and the undo's `restoreDefinition` (`internal/stacks/update.go`, `undo.go`; noted 2026-10-24)* |
|
||||
| the restore (`stackAdapter.RecreateStackDefinitionFromUnit`) | the recovery unit's captured compose — **this closes R-441** |
|
||||
| `Manager.AdoptPins` | the observation, once, and only when complete AND matching |
|
||||
|
||||
@@ -1998,7 +2101,7 @@ earlier feature is the failure mode to look for whenever a file changes meaning.
|
||||
| **3** | **The compose file becomes DERIVED** — the pin in `app.yaml` wins; the syncer renders instead of copying. | **SHIPPED, controller v0.235.0 (2026-09-06)** — operator ruling §3.4 |
|
||||
| **4** | **A guarded update** — verified-backup precondition, abort-on-failure, and the truth at the moment of action rather than 5m16s later (R-443). | **SHIPPED + PROVEN LIVE, controller v0.237.0 (job) + v0.238.0 (page) + v0.238.1 (2026-09-13); any backup tier since v0.239.0 (§3 decision 8)** — §6.1 |
|
||||
| **5** | **An upgrade test that runs again** — a harness that upgrades a real app with real data in it and asks the app for the data back. | **SHIPPED, `app-catalog/scripts/upgrade-test.py` (2026-09-06)** — 7 edges, 3 apps; see §4.1 and §10 |
|
||||
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | OPEN — R-450; rulings 2026-09-23; build order §6.4 |
|
||||
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | ~~OPEN — R-450~~ **BUILT** — the leg in `internal/stacks/unattended.go`, wired into the off-site job; R-450 CLOSED 2026-10-05 (capability map B11) *(noted 2026-10-24, review round 2)* |
|
||||
| **7** | **A fleet sweep pipeline** — the operator can see, and move, how far behind every box is. | OPEN — R-451; ruled 2026-09-23 (decision 18), built later |
|
||||
|
||||
### 6.1 Slice 4 as SHIPPED (controller v0.237.0 + v0.238.0, 2026-09-13)
|
||||
@@ -2013,12 +2116,13 @@ closed by construction: nothing reports an update complete on the compose exit c
|
||||
| # | phase | what happens | on failure |
|
||||
|---|---|---|---|
|
||||
| 0 | refusals (409, before the intent is recorded) | held (R-439), busy (backup/restore/app-data op/quiesce), migration, already updating, deploying, memory (the deploy's `memoryVerdict`, releasing the app's own request), disk (**fixed 2 GB floor** — image size unknown without a registry), **no copy on ANY tier and no backup can be taken now** (since v0.239.0; before it, no restorable Tier-2 copy) | nothing moves, nothing is recorded |
|
||||
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) | nothing moves |
|
||||
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) — *an app whose data lives in bind-mounted files walks second drive → OFF-SITE → own unit (decision 9, R-479, v0.241.0; `internal/backup/update_guard.go` `updateTierOrderBindData`; noted 2026-10-24)* | nothing moves |
|
||||
| 2 | `backing-up` — only when no tier holds a fresh copy | `RunAppBackupNow`: this app's DB dump → volume dump → unit capture (marked proven current) → Tier-2 copy, whose failure is a WARN since v0.239.0 | refused with the backup's own error; nothing moves |
|
||||
| 3 | `safety-dump` | `WriteUpdateSafetyDump` (R-361's undo copy) — **before the pin moves** | refused; nothing moves |
|
||||
| 4 | `pinning` | the previous definition is copied aside and journaled, then the pin advances | pin put back |
|
||||
| 5 | `pulling` | `compose pull` | **pin and definition PUT BACK** — nothing ran (Scenario E) |
|
||||
| 5a | `copying` **(v0.263.0)** | `compose stop`, then each NAMED volume `cp -a` into `<volume>.pre-update-<stamp>` by a helper that writes a finished-marker last (bind folders never) | copies removed, **pin put back, the old version started again** — nothing new ran |
|
||||
| 5b | `converting` **(v0.273.0, row added 2026-10-24)** | only on a step whose ladder entry carries `engine_conversion`: the box converts the PostgreSQL major (dump from the old engine, load into the new, check) — `internal/stacks/pgconvert.go` | the undo puts the old engine's data back |
|
||||
| 6 | `starting` | `compose up -d --remove-orphans` | **UNDO** (below), HOLD only if the undo fails |
|
||||
| 7 | `verifying` | the `.felhom.yml` health check through the existing probe **when it resolves to a container**, else 60 s of every container running and none restarting; bounded by `update.health_timeout` | **UNDO** (below), HOLD only if the undo fails. Before v0.263.0: stop + HOLD, the pin stays (Scenario F) |
|
||||
| 7a | `undoing` **(v0.263.0)** | every copy validated (marker) BEFORE anything is poured back; volumes emptied and refilled; definition, pin and the pinned version's `.felhom.yml` record put back from the job's own copies; `up`; health with the OLD version's probe | HOLD, the sentence prefixed *„A frissítés nem sikerült, és az automatikus visszaállítás sem."* + the data state (`untouched` / `half` / `not_started`); copies kept |
|
||||
@@ -2067,7 +2171,7 @@ the harness has proven it, is slice 6's.
|
||||
> copy taken before the pin moved**, so the old version meets pre-migration data. Spiked by hand on
|
||||
> 9202 the same day — §6.1a.
|
||||
|
||||
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6.
|
||||
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6. *(Noted 2026-10-24: stepping is built — the ladder, one tested step per press, decision 14.)*
|
||||
|
||||
### 6.1a The undo (decision 15) — SPIKED BY HAND, then BUILT: controller v0.263.2 (2026-09-23)
|
||||
|
||||
@@ -2081,7 +2185,8 @@ only the live box could show, fixed the same day:** the undo's probe was never a
|
||||
current probe held the app `unhealthy` (v0.263.1), and the "old" `.felhom.yml` taken at update time
|
||||
was already the new one, because `.felhom.yml` flows in on every catalog sync (v0.263.2: the pinned
|
||||
version's file is now recorded in `applied-meta/` whenever a version is pinned). **Residual (R-646):**
|
||||
an app pinned before v0.263.2 has no such record until its next pin. **Two more, found by the chaos hour of
|
||||
an app pinned before v0.263.2 has no such record until its next pin. *(Closed since: `BackfillAppliedMeta` records it at
|
||||
start for every app CURRENT with the catalog, and names the rest — `internal/stacks/undo.go:646-652`; noted 2026-10-24.)* **Two more, found by the chaos hour of
|
||||
2026-09-23 night (`audits/DRILL-night-2026-09-23.md` Part D):** the undo finds an app's volumes by their compose
|
||||
label, and a restore recreates them WITHOUT it — so after any restore the undo copies nothing (R-658, P1); and a
|
||||
held file-leg app is pointed at a restore that refuses a database-only copy, with no route left on a box without
|
||||
@@ -2221,7 +2326,7 @@ breaks.
|
||||
|
||||
---
|
||||
|
||||
### 6.2 Slice 6, as it will be built (OPEN — R-450; ruled 2026-09-23, decisions 11–17)
|
||||
### 6.2 Slice 6, as it will be built (~~OPEN — R-450~~ BUILT, R-450 closed 2026-10-05 — noted 2026-10-24; ruled 2026-09-23, decisions 11–17)
|
||||
|
||||
**The shape the rulings fix. The build order and costs are §6.4.** Rewritten 2026-09-23; the
|
||||
earlier "as it would be built" draft keyed on the window of Q1's recommendation and on
|
||||
@@ -2292,7 +2397,7 @@ retried. A working caller in this exact shape exists as evidence, not product:
|
||||
**The per-box switch key is `app_update.unattended`**, default **true** (decision 12).
|
||||
|
||||
⚠ **NOT `auto_update` — that name is TAKEN, and by the very thing this must not collide with.**
|
||||
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281, default
|
||||
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281 then, L278-279 now; default
|
||||
**04:30** at L422) are the CONTROLLER's own update. Two settings with that name, one meaning the
|
||||
controller and one meaning apps, is the kind of collision that is only discovered by an operator who
|
||||
turned off the wrong one. The app-scoped key is `app_update.*`. **And `stacks.update_window` is
|
||||
@@ -2322,12 +2427,12 @@ what the part can do to a household's data if it is wrong, not how likely that i
|
||||
|
||||
| # | part | rulings / rows | cost | depends on | risk to customer data |
|
||||
|---|---|---|---|---|---|
|
||||
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold`: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
|
||||
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold` *(as SHIPPED — noted 2026-10-24: the copy is the FOLDER copy decision 19 chose: every named volume `cp -a` into a sibling volume with a finished-marker, put back on failure; no database load — `internal/stacks/undo.go:26-36`; the text that follows is the design before the bake-off)*: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
|
||||
| **2** | **SHIPPED — controller v0.264.0 + hub v0.120.0, proven live on 9202 and 9201 2026-09-23** (`audits/undo-fleet-2026-09-23/`). **The update sentences in the household's language** (R-606) and a mail when an automatic update is undone or held: events `app_update_undone` (warning) and `app_update_held` (error), one per app per outcome, on by default, mailed in the household's language with the app named in the subject; per-app cooldown on both legs. Leftovers: R-647. | R-606, 15 | **1** | — | none |
|
||||
| **3** | **SHIPPED — controller v0.264.0, proven live on 9202 2026-09-23.** **A disabled notifier says so** (R-620), so the mail of part 2 can be measured on a scratch box at all. | R-620 | **0.5** | — | none |
|
||||
| **4** | **SHIPPED — catalog `6db08a5`, night 2026-09-23** (`audits/DRILL-night-2026-09-23.md`): `update_ladder:` in `.felhom.yml`, one JSON entry per line (spiked on controller v0.266.0 and v0.267.0 first — the controller ignores the key); `check-test-record.py` (static, CI) + `check-test-record-move.py` (history + the registry for moved refs only, decision 23); the only writer `upgrade-test.py --write-ladder`; the 21 moves of 2026-09-22 backfilled from their records (21 proven). Not built: `steps/<to>.yml` (part 5 needs it once an app has two steps); `CompareImageRefs` did NOT move to the gate — the gate asks for a proven test instead, which is decision 13's own test. **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only |
|
||||
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
|
||||
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. **The box half (compare, render `name:tag@sha256`) is not built.** **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
|
||||
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). *(Closed since: a step's own `steps/<key>.felhom.yml` is read when present — `ladder.go:127-132`, v0.269.0; noted 2026-10-24.)* **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
|
||||
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. ~~**The box half (compare, render `name:tag@sha256`) is not built.**~~ *Built — the box renders the tested digest, carries it on sync, and the badge reads the tested digest (`internal/stacks/digest.go:15-24`, `updateorder.go`; noted 2026-10-24).* **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
|
||||
| **7** | **SHIPPED — controller v0.271.0 (`9cf13a3`), proven on 9202 over four simulated nights 2026-09-24/25 and watched through the demo boxes' first real automatic night (`audits/DRILL-night-2026-09-25.md`).** Decisions 31–33 taken unattended. Gaps a scratch box cannot close: R-687; no resume after a restart: R-686. *(Before:)* **SPIKED 2026-09-24 (night, Part C) — the measured build brief is §6.4.2 below.** **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has ENDED (on every exit path, skips included), one app at a time, one step per press, `app_update.unattended` default ON (decision 12), `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it, and the full-system backup's gate waits for it until W+5h (decision 20). | 11, 12, 14, 15, 20 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe |
|
||||
| **8** | **SHIPPED — controller v0.265.0 + hub v0.121.0, proven live on 9202 2026-09-23** (`audits/cleanup-2026-09-23/`; a kernel `oom_kill` counter, not the sticky flag — `08` §6.2). **R-636** — the same OOM key re-firing escalates instead of staying one `warning` for six hours. | R-636 | **1** | — | none |
|
||||
| **9** | **SHIPPED — controller v0.265.0, proven live on 9202 2026-09-23** (badge „Megállítva — visszaállítás szükséges" / "Stopped — restore needed", no Update button, 409 `held` unchanged). **R-625** — a held app stops offering an Update it will refuse. With the undo, holds become rarer; the lie on the page does not go away by itself. | R-625 | **0.5** | 1 | none |
|
||||
@@ -2410,7 +2515,7 @@ During the db-dump leg every press was refused `busy` (transient, retried next p
|
||||
update's own finish should refresh them.
|
||||
- **An Update pressed on an app already at the head runs the whole guarded update** — backup, pull,
|
||||
restart, `done` — with no refusal (R-679: navidrome restarted four times for nothing). The leg never
|
||||
presses an app whose pin equals the head; the preflight should refuse with reason `current`.
|
||||
presses an app whose pin equals the head; the preflight should refuse with reason `current` (as built the reason key is `already_current`, `update.go:432` — noted 2026-10-24).
|
||||
- **The product does not remember a failed step.** After vikunja's undo the badge still offers the
|
||||
same step; only the caller's memory kept it from a re-press. Decision 15 needs it on the box: the
|
||||
failed `to` is recorded per app and the leg skips it until the catalog's ladder changes.
|
||||
@@ -2492,6 +2597,24 @@ and E are the ones that unblock a decision; leg A is the one that takes the time
|
||||
harness on DooPlex for the image-side ones.
|
||||
|
||||
|
||||
### 6.4.x Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
|
||||
|
||||
Read at `file:line` on controller `main` (`321df1f`) by the main session:
|
||||
|
||||
- **`build.sh --push` also refuses a DIRTY tree, a HEAD that is not `origin/main`, and a directory that is not a git
|
||||
checkout** — `controller/build.sh:144-166`. §0 names only the tag-exists and unreachable-registry refusals.
|
||||
- **The leg retries a passing refusal ONCE in the same night** (`busy`, `updating`, `deploying`, `migrating`,
|
||||
`self_updating`), after `RetryWait` — not „the next pass/night" (`internal/stacks/unattended.go:65-66`, the pass loop).
|
||||
- **The leg skips an app the household stopped** (`stopped_by_household`, R-721): an update would start it
|
||||
(`unattended.go:416-418`).
|
||||
- **A manual run of the chain** (R-705, the debug action) gives the leg a deadline of its start + the leg's normal
|
||||
length (`backupwindow.UpdateLegLengthMin`), not W+5h (`unattended.go:232-238`).
|
||||
- **Two preflight refusals are plain skips to the leg:** `already_current` and `engine_no_test`
|
||||
(`internal/stacks/update.go:432,454`).
|
||||
- **The render table (§5.4) has one more row:** when the catalog template's images cannot be read, the syncer leaves
|
||||
the app's compose file alone for that cycle (`internal/sync/sync.go:622-627`); and the box's R-971 allow-list runs
|
||||
before ANY compose write (`cmd/controller/main.go:421`, `SetAdmitFn`).
|
||||
|
||||
## 6.5 The drill catalog and the image store — the standing method for update drills
|
||||
|
||||
**Why this section exists.** On 2026-09-21 an afternoon session put a deliberately broken image into
|
||||
@@ -2580,7 +2703,9 @@ customer box has credentials for it. The store listens on the guest's loopback o
|
||||
### 7.1 The record (slice 1)
|
||||
|
||||
`Manager.recordInstalledImages` (`felhom-controller/controller/internal/stacks/installed.go`) runs
|
||||
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`,
|
||||
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`
|
||||
*(as built — noted 2026-10-24: `UpdateStack` is gone; the callers are `StartStack`, `RestartStack`, the deploy path, the
|
||||
end of the guarded update and the end of the undo — `manager.go`, `deploy.go`, `update.go`, `undo.go`)*,
|
||||
and writes `app.yaml`:
|
||||
|
||||
```yaml
|
||||
@@ -2656,7 +2781,7 @@ Version strings stay in the logs, the API and the hub.
|
||||
The 8th, immich's own ghcr build, is UNMEASURED — ghcr exposes no anonymous last-modified. So on
|
||||
demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.
|
||||
Digest-level comparison needs the catalog to record the digest at push time — **R-446**, put to the
|
||||
operator as §3b **Q6**, recommended YES.
|
||||
operator as §3b **Q6**, recommended YES. *(Built since for every ref with a ladder entry — `updateorder.go`, `digest.go`; the limitation remains only for a ref no entry records; noted 2026-10-24.)*
|
||||
2. **~~Nothing enforces `catalog_since`.~~ Enforced by the pre-push hook since 2026-09-13 (R-452, `app-catalog-felhom.eu/scripts/check-catalog-since.py`); CI's shallow clone still skips it out loud.** A commit that moves an `image:` line and forgets the date
|
||||
under-reports how far behind a box is. The gates runner fetches at `--depth 1` and has no parent
|
||||
commit to diff against, so the gate needs a deeper fetch — **R-452**.
|
||||
@@ -2681,13 +2806,18 @@ Version strings stay in the logs, the API and the hub.
|
||||
refuses without a restorable, proven Tier-2 copy, backs up first when that copy is stale, takes a
|
||||
safety dump, and holds an app that does not come up. **What stays true:** it still has no automatic
|
||||
rollback (deliberately, §6.1) and can still attempt a multi-major jump the app will refuse (R-40) —
|
||||
that now ends HELD rather than crash-looping behind a green button.
|
||||
that now ends HELD rather than crash-looping behind a green button. *(Dated note 2026-10-24: a copy on ANY tier now
|
||||
satisfies the precondition (v0.239.0), the box undoes a failed update itself (decision 15, v0.263.2), and the
|
||||
ladder climbs one tested step per press (decision 14) — `internal/backup/update_guard.go`, `internal/stacks/undo.go`,
|
||||
`ladder.go`.)*
|
||||
7. ~~**An engine major can be applied without its datadir upgrade, and nothing notices.**~~ **CLOSED
|
||||
2026-09-13 for MariaDB (R-459):** every `mariadb:` sidecar carries `MARIADB_AUTO_UPGRADE=1`, and the
|
||||
harness shows the conversion running on the E3/E3b edges (§3 decision 5). **What stays true:** the
|
||||
PostgreSQL half (R-463) has no equivalent — the image performs no `pg_upgrade` — and the
|
||||
engine-major rule (§3 precaution 3, R-469) is what keeps both engines inside their major until
|
||||
Slice 4 gives the Update button a backup.
|
||||
Slice 4 gives the Update button a backup. *(Dated note 2026-10-24: MariaDB is lifted — its major is its own edge; a
|
||||
PostgreSQL major may pass with a two-venue proof and an `engine_conversion` mark, and the box converts it
|
||||
(v0.273.0) — `app-catalog-felhom.eu/scripts/check-engine-major.py`.)*
|
||||
8. ~~**Only three of 53 apps have ever had an upgrade measured.**~~ **WIDENED 2026-09-21 to 21
|
||||
EDGES ACROSS 19 APPS** (`audits/DRILL-update-night-2026-09-21.md`), on scratch guest 9202
|
||||
through the product's own guarded Update, against a PRIVATE DRILL CATALOG so the live catalog
|
||||
|
||||
@@ -128,7 +128,9 @@ the hub's e-mails follow.**
|
||||
`expired` pinned to Hungarian so the language cannot become the oracle the text refuses to be.
|
||||
Full design: `05-hub-architecture.md` §15. **R-555 closed** — the `language` allowlist entry is out
|
||||
of `wire_contract_gate.py` and the gate now checks the field for real.
|
||||
- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece:** the hub stores a per-customer language, renders it into
|
||||
- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece** *(Part B is built
|
||||
too — the controller reads `customer.language` as its start language since v0.256.0, `internal/config/config.go:149-154`;
|
||||
corrected 2026-10-24)*: the hub stores a per-customer language, renders it into
|
||||
`controller.yaml` next to `customer.id/name/domain/email` (`hub/internal/configgen/configgen.go`),
|
||||
and the box uses it **only while the household has never chosen** (`settings.json` empty). The
|
||||
household's own choice always wins; the hub's e-mails follow the language the box **reports**, which
|
||||
@@ -184,7 +186,8 @@ bundles. They now judge the page **as rendered in each language** (`controller/s
|
||||
- **hu** — the product speaks in „te". The converted copy carries formal („ön") forms (R-516). They are
|
||||
not fixed by a localisation release (§1), so the gate counts them against `HU_FORMAL_CEILING` as a
|
||||
ratchet: a new one convicts, fixing one lowers the ceiling. The ceiling followed the conversion:
|
||||
6 (spike) → 10 (release A) → 12 (release B) → **16** (release C). **[FACT] Measured limit:** the stem
|
||||
6 (spike) → 10 (release A) → 12 (release B) → **16** (release C) → … → **0** (R-516 item 12, the last eleven in the
|
||||
te-form — `felhom-controller/controller/scripts/i18n_missing_gate.py:41`; corrected 2026-10-24). **[FACT] Measured limit:** the stem
|
||||
list is short — on the release C pages it counts 4 formal forms, while a wider list („írja be",
|
||||
„adja meg", „válassza", „biztosan eltávolítja", „engedélyezze", …) finds at least 22 keys (R-516).
|
||||
- **en** — second person, plain: no „please", no „kindly".
|
||||
@@ -201,7 +204,8 @@ vissza") — R-564. The hub copy gate and the catalog have no language concept
|
||||
- **[DESIGN] The hub's operator pages** — the operator reads them; they stay English (decision 1).
|
||||
- **[DESIGN] The first-boot wizard** (`controller/internal/setup/`, 8 pages, 95 strings) — out of scope
|
||||
and obsolete (decision 4, 2026-09-17; `02-controller-module-map.md` L56). A household still reaches
|
||||
it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**.
|
||||
it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**. *(Deleted since:
|
||||
`internal/setup/` is gone (v0.308.0); `bootstrap.WaitForConfig` shows a waiting page instead — corrected 2026-10-24.)*
|
||||
- **[FACT]** App containers' own UIs (Uptime Kuma, PrivateBin…) — not Felhom's copy (R-516 items 5-6).
|
||||
|
||||
---
|
||||
@@ -226,7 +230,7 @@ i18n:
|
||||
```
|
||||
|
||||
Why: one app's copy stays in one file a reviewer sees whole; the controller's `yaml.Unmarshal` ignores
|
||||
unknown keys (`internal/stacks/metadata.go` L316 — **verified: this repo constructs no `yaml.Decoder`
|
||||
unknown keys (`internal/stacks/metadata.go` L316 then, L475 now — **verified: this repo constructs no `yaml.Decoder`
|
||||
at all, so there is no `KnownFields` anywhere to turn strict**), so older controllers are unaffected;
|
||||
a missing English field falls back field by field.
|
||||
|
||||
@@ -234,7 +238,7 @@ a missing English field falls back field by field.
|
||||
reached only through `LocalizeStacks` / `LocalizeStackPtr` / `Stack.MetaFor`.
|
||||
|
||||
- **`For("hu")` is the parsed struct with `I18n` cleared and nothing else**, deep-compared against all
|
||||
53 real catalog files (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
|
||||
53 real catalog files (the identity fixtures are still those 53; the catalog has 65 apps — noted 2026-10-24) (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
|
||||
§1 therefore holds by construction for catalog copy, and `LocalizeStacks` returns the INPUT SLICE
|
||||
for Hungarian rather than a copy, so nothing on the Hungarian path is even touched.
|
||||
- **Fallback is field by field, and blank counts as absent** (ruling 3). A half-translated app is a
|
||||
@@ -307,10 +311,10 @@ for the English page test (R-565), arriving again in a different repo. So
|
||||
`app-catalog-felhom.eu/scripts/check-copy-i18n.py` folds and matches word-bounded stems, with a
|
||||
positive and two negative controls printed on every run.
|
||||
|
||||
**[FACT] The gate**, fifth row of `catalog_gates.py`, static, in the pre-push hook:
|
||||
**[FACT] The gate**, fifth row of `catalog_gates.py` (the sixth since — corrected 2026-10-24), static, in the pre-push hook:
|
||||
|
||||
1. **FREEZE** — every Hungarian copy string equals `scripts/copy_freeze/hu.json`, captured in its own
|
||||
commit before any translation. Runs on all 53 apps whatever scope is named. A new app must be
|
||||
commit before any translation. Runs on every app in the freeze file (65 on 2026-10-24) whatever scope is named. A new app must be
|
||||
admitted with `--add-app NAME --reason "…"`.
|
||||
2. **STRUCTURE** — the `en` block may carry copy fields and nothing else; every key-matched entry must
|
||||
have a Hungarian twin, or it is INERT on the box and the translator never learns.
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
|
||||
> | | |
|
||||
> |---|---|
|
||||
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1).** Ratification is Viktor's review, not an editor's. |
|
||||
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1); steps 2–6 are BUILT by §8's own list (corrected 2026-10-24 — the ratification is still the operator's).** Ratification is Viktor's review, not an editor's. |
|
||||
> | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. |
|
||||
> | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 |
|
||||
> | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. |
|
||||
@@ -76,10 +76,14 @@ boxes installed. Nobody keeps a hand-written list.
|
||||
- **A BYO host** (the owner brought their own Proxmox). `[FACT]` The installer leaves its repositories
|
||||
alone: *"apt repo alignment skipped (byo — the owner manages repos)"* (`scripts/felhom-host-install.sh`,
|
||||
`align_apt_repos`). `[PROPOSAL]` On a BYO box we update the guest and Docker only, never the host.
|
||||
**[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased]** The host's root-key-only sshd
|
||||
drop-in (D4, R-982) is an APPLIANCE-only file of the config bundle: written only when the root-owned install record
|
||||
says `appliance`; on a `byo` record nothing is written and a copy the bundle wrote earlier (this path, our first line)
|
||||
is removed; a box whose record proves nothing is left as it is (R-1257).
|
||||
- **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5).
|
||||
- **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in
|
||||
the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since
|
||||
2026-06-11) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
|
||||
2026-06-11 — *`2026.9.3` today, corrected 2026-10-24*) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
|
||||
— the app-image question (`09`), not an OS package. The gap is R-838.
|
||||
|
||||
---
|
||||
@@ -101,9 +105,10 @@ gets the Docker engine that was current when its golden was baked, and keeps it.
|
||||
|
||||
~~**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist
|
||||
(`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`.~~
|
||||
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:58`
|
||||
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:107`) and `:194`
|
||||
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:628`). Nothing else.
|
||||
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:68`
|
||||
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:133`) and `:228`
|
||||
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:659`). Nothing else. *(Line numbers
|
||||
corrected 2026-10-24.)*
|
||||
`apt` as root runs package scripts as root, so a broad `apt` grant is a full root grant. §5.4
|
||||
proposes how to avoid that.
|
||||
|
||||
@@ -250,7 +255,7 @@ approval of that day stays in force (decided by CC unattended — operator may r
|
||||
versions. A broken-into hub can choose an older version or no version. It cannot make a box install a
|
||||
package that the publisher did not sign.
|
||||
|
||||
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]`
|
||||
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]` — *BUILT and installed since (§8.1); the agent's sudoers holds only the `--plan` line, no `--repair-only` (`felhom-agent/configs/felhom-agent.sudoers:337`); noted 2026-10-24*
|
||||
|
||||
`felhom-os-apply` — root-owned (`0755 root:root`), installed by the installer beside `felhom-selfupdate-guarded`;
|
||||
the agent's sudoers gets exactly `/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json` and
|
||||
@@ -315,7 +320,7 @@ PID 1 or `lxc-start`, or a kernel was installed.
|
||||
Decision 96. Before it, the signed `agent_update` replaced only the binary; wrappers, units and sudoers lines reached
|
||||
an installed box by reinstall or by hand.
|
||||
|
||||
- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22: sudoers ×2, the
|
||||
- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22 then, **32** counted 2026-10-24 — plus the vzdump-partial-clean, thin-check, internal-place and priv-apply wrappers, the guest hook, the shared-parent unit, the GRUB one-shot pair and the sshd root-key-only drop-in; sudoers ×2, the
|
||||
five wrappers, the crash guard and its units + config, the agent and rollback units, the start-limit drop-in, the mgmt
|
||||
watchdog and its tmpfiles/units, the OOB belt's four files). `scripts/build-config-bundle.py` builds the bundle from it
|
||||
reproducibly; `release-agent.sh` publishes it beside the binary; the hub vouches its sha with the agent (exact-name
|
||||
@@ -393,6 +398,9 @@ must never overlap a backup, a restore-test or a self-update.~~
|
||||
- **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release,
|
||||
how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest
|
||||
approved release raises an alarm (`08`).
|
||||
- **[FACT, added 2026-10-24 — hub v0.169.0] The System page's „Upload line" cell** shows the best of the last SUCCESSFUL
|
||||
weekly line test (R-1247), amber when it is older than 15 days or was never measured after an attempt — judged on
|
||||
`last_ok_at`, never on the attempt time (`hub/internal/web/system.go` `lineStaleAfter`, `lineTestCell`).
|
||||
- **[FACT, hub v0.132.0] The System page** (`/system`, R-852, decision 89): one row per box — ring and switch with
|
||||
buttons, the tunnel, host Proxmox / running and next-boot kernel / Debian / release / pending / not covered / held /
|
||||
reboot needed / `kernel.panic` / oops / crash restarts / the guard, guest Debian / release / pending / restart needed,
|
||||
@@ -480,7 +488,9 @@ Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-do
|
||||
**What it updates:** the host's Proxmox USERSPACE packages (pve-manager, qemu-server, pve-container, lxc-pve,
|
||||
libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 packages behind on the demo boxes on
|
||||
2026-10-07. **What it never touches:** a kernel, boot, firmware or microcode name (`HOST_SLOW_RE`: `proxmox-kernel*`,
|
||||
`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`)
|
||||
`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`,
|
||||
and — read 2026-10-24 in `felhom-os-apply` — `linux-(image|headers|kbuild|modules|base)*`, `firmware-*`,
|
||||
`proxmox-secure-boot-support`)
|
||||
— that is R-836's lane (decision 164). No reboot.
|
||||
|
||||
- **Wrapper** (`felhom-agent/configs/felhom-os-apply`): layer `pve`, lane `slow` only. Origin `Proxmox Debian
|
||||
|
||||
@@ -20,8 +20,10 @@ workspace-root `CLAUDE.md` § „Artifact taxonomy". **[FACT]** The spec shape i
|
||||
class). Validation is project Claude's unless CC is asked (`CLAUDE.md`, same section).
|
||||
|
||||
**A session with no spec** (a `/goal`, a night brief, „work the register") inherits the discipline a
|
||||
spec used to carry from `.claude/rules/unprompted-work.md` — **[FACT]** five byte-identical copies:
|
||||
the workspace root and each of the four repos (the file's own header says „change all five or none").
|
||||
spec used to carry from `.claude/rules/unprompted-work.md` — **[FACT]** five copies:
|
||||
the workspace root and each of the four repos (the file's own header says „change all five or none"). *(Measured
|
||||
2026-10-24: they are NOT byte-identical — the felhom.eu and felhom-controller copies carry one more paragraph, „A
|
||||
version number is pushed once" (added 2026-10-15); the review round 2 session aligned the other three, see its report.)*
|
||||
|
||||
## 2. Where an instruction lives
|
||||
|
||||
@@ -29,7 +31,7 @@ the workspace root and each of the four repos (the file's own header says „cha
|
||||
|---|---|---|
|
||||
| Cross-repo facts and standing rules (production host, gates, secrets, CHANGELOG/REPORT) | workspace-root `CLAUDE.md` | every session in the workspace |
|
||||
| One repo's stable orientation | `<repo>/CLAUDE.md` | when a file in that repo is touched |
|
||||
| Path-scoped rules (a subsystem's traps) | `<repo>/.claude/rules/*.md` | when a matching path is touched |
|
||||
| Path-scoped rules (a subsystem's traps) | `<repo>/.claude/rules/*.md` | when a matching path is touched — a file with `unconditional: true` (e.g. `live-probes.md`) or without `paths:` (the `unprompted-work.md` copies) loads every session (corrected 2026-10-24) |
|
||||
| A procedure with steps (build, test, diagnose, write for the operator) | a skill, `felhom.eu/skills/<name>/SKILL.md` | when its trigger matches, or by name |
|
||||
| Current state, decisions, open work | `CONTEXT.md`, `STATUS.md`, `documentation/backlog/OPEN-ITEMS.md` | read on demand — never copied into an instruction file |
|
||||
| Durable cross-session facts about the operator and the machines | the memory folder (`.claude-memory/`, index `MEMORY.md`) | the index each session; files on recall |
|
||||
@@ -38,8 +40,9 @@ Which rung a piece of material belongs on is the doc-authoring skill's §3 („W
|
||||
material sits") and §6 („One rule, one home"). This page does not repeat them.
|
||||
|
||||
**[FACT] Skills** are written in `felhom.eu/skills/`, installed by
|
||||
`python3 felhom.eu/scripts/install_skills.py` as symlinks into `~/.claude/skills/` (so a repo edit is
|
||||
live at once), and checked by `python3 felhom.eu/scripts/check_skills.py` (shape, the 150-line limit
|
||||
`python3 felhom.eu/scripts/install_skills.py` as symlinks into `~/.claude/skills/` (so an edit to an INSTALLED skill is
|
||||
live at once; a NEW skill folder is live only after `install_skills.py` runs again — measured 2026-10-24:
|
||||
`skills/felhom-web-design` has no link yet), and checked by `python3 felhom.eu/scripts/check_skills.py` (shape, the 150-line limit
|
||||
and its one grandfathered entry, R-394). The list is the folder; `skills/SOURCES.md` names where each
|
||||
skill's material came from.
|
||||
|
||||
|
||||
@@ -36,6 +36,7 @@ decision (R-925).
|
||||
| g | a compromised Gitea / registry / build | push to the catalog, the images and the installer tag |
|
||||
| h | a thief with the box | the disks |
|
||||
| i | Cloudflare | the plaintext of tunnel traffic — **accepted** (R-904) |
|
||||
| j | **added 2026-10-23:** a hacked homelab app pod on DooPlex (happened: Plex2Shell, HM-040) | the pod network with free egress: every cluster Service, the node's ports (NFS, kubelet, k3s API), the LAN masqueraded as DooPlex; no service-account token |
|
||||
|
||||
## 4. Entry points
|
||||
|
||||
@@ -46,8 +47,8 @@ scripts are held with the review's working notes.
|
||||
| component | routes | no credential | listeners |
|
||||
|---|---|---|---|
|
||||
| controller | 231 (+8 in setup mode) | 21, of which 7 change state, each behind its own password, code or token | port 8080 on the guest's Docker networks (not published); mail relay 2525/2465/2526 |
|
||||
| hub | 121 (1 health, 53 API, 67 web — v0.152.0 added the two token routes; both route tables are now pinned to the source by tests, R-952) | appliance register (rate-limited, by design), re-enroll (only while the operator has armed recovery mode), login, health | one ingress, path `/`, geo filter (HU, IE, LAN) |
|
||||
| agent | 38 local-API routes (`POST /host/poweroff` added 2026-10-13, R-998) | 0 | local API on the island bridge; WireGuard; poke (tunnel only); felhom-sshd (tunnel only, measured) |
|
||||
| hub | 121 (1 health, 53 API, 67 web — v0.152.0 added the two token routes; both route tables are now pinned to the source by tests, R-952) — *the API table has 54 entries since `/box-config/` (v0.161.0) and `/preferences` (v0.165.0); counted from `hub/internal/api/routes.go`, 2026-10-24* | appliance register (rate-limited, by design), re-enroll (only while the operator has armed recovery mode), login, health, and the public self-bind page `/bind/<token>` (a capability link, added to this list 2026-10-24) | one ingress, path `/`, geo filter (HU, IE, LAN) |
|
||||
| agent | 41 local-API routes on `main` (counted 2026-10-24 in `felhom-agent/internal/localapi/server.go`; `POST /host/poweroff` added 2026-10-13, R-998; `POST /host/line-test/run` added 2026-10-24, R-1250, unreleased) | 0 | local API on the island bridge; WireGuard; poke (tunnel only); felhom-sshd (tunnel only, measured) |
|
||||
|
||||
**Outbound fetches that bring code or config onto a box:** the catalog (git, `main`), app images, the four infra
|
||||
images, the controller image (by tag), `controller.yaml` from the hub, the agent binary and config bundle (signed),
|
||||
@@ -57,11 +58,11 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
|
||||
|
||||
| boundary | verdict | evidence |
|
||||
|---|---|---|
|
||||
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981) |
|
||||
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981). **Re-check (2026-10-24, review round 2):** the socket is gone from the homepage template (catalog `templates/homepage/docker-compose.yml:21`) and the box's allow-list is empty, so a socket bind is refused (`felhom-controller/controller/internal/privpolicy/allowlist.json`) |
|
||||
| **added 2026-10-14 (R-1033, P1):** a visitor ↔ Cloudflare over plain http | holds on boxes with controller ≥ 0.312.0 | before: every public name answered `http://` (login pages too), and Sonarr/Radarr/Prowlarr redirected https visitors to an absolute `http://` login; a password could cross an untrusted network unencrypted (attacker: anyone on the visitor's network). Now traefik sends every `http://` request to https (301) before an app sees it, HSTS one day; boxes below 0.312.0 (Tester 2 at 0.292.0) keep the gap until the floor delivers it. `audits/https-only-2026-10-14/` |
|
||||
| **added 2026-10-15 (R-1045):** a stranger ↔ a fresh Plex | holds on controller ≥ 0.313.0; on older ones only while the household does not press „Kész" before claiming | an unclaimed Plex answers everything to a private address with no token, and through the tunnel every request comes from one. The setup gate (catalog `9a23ee5`) closes the public path; 0.313.0 opens it only when Plex itself says claimed. A status probe meant to cover older controllers OPENED an unclaimed Plex while it started (`audits/media-build-2026-10-15/G7/`) and was removed |
|
||||
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976) |
|
||||
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978), and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
|
||||
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976). **Re-check (2026-10-24, review round 2):** both fixed — a password change or a claim/reset ends every dashboard, family and gate session (controller v0.324.0, `internal/web/auth.go:352-366`) |
|
||||
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978) *(re-check 2026-10-24: both narrowed — a cadence cap on `POST /backup` since agent v0.160.0, `internal/localapi/backup_cap.go`; a new escrow package is uploaded only at the claim since agent v0.160.0, `escrow_ceremony.go`)*, and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
|
||||
| **added (R-1007):** controller → an app's own job queue | holds — nothing new | the `after_unclean_boot` commands run inside the app's container through the docker socket the controller already holds, and read the app's database with a read-only session: no key into the household's account, nothing a compromised controller could not already do (exec in any app, read its database). The household's lost-file names stay on the box (`app.yaml`, the app page); the operator event carries counts only, and a failing command's output is never logged (security review of controller `8d862e4`, fixed in `3b85a0c`) |
|
||||
| **added:** agent ↔ root on the host | holds by the sudoers | every root verb is one regex-anchored line or a root wrapper that re-checks its input (`03` §3); agent v0.156.1 adds one delete: `felhom-vzdump-partial-clean` removes only a DEAD half-written vzdump of the named guest on a `dir` storage (no vzdump on the host, nothing holding it open) — a compromised agent gains the deletion of a leftover that is not a backup |
|
||||
| agent ↔ hub | holds for destructive ops | signed jobs verified in the documented order; unsigned spec has no size bounds (R-984) |
|
||||
@@ -69,12 +70,21 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
|
||||
| controller ↔ hub | **did not hold until 2026-10-12** | the global key was public (R-950, rotated) and three routes needed no key (R-951, fixed); per-box keys are bound to their box (R-955) |
|
||||
| box ↔ PBS | holds as written | ciphertext only; the zero-knowledge wording is stronger than a live box allows (R-985) |
|
||||
| guest ↔ Proxmox host | holds | the guest has no Proxmox credentials (measured) |
|
||||
| hub ↔ Cloudflare API | holds for the zone check | tokens are stored unsealed in the hub database (R-975); R-137 open |
|
||||
| hub ↔ Cloudflare API | holds for the zone check | tokens are stored unsealed in the hub database (R-975); R-137 open. **Re-check (2026-10-24, review round 2):** the tokens are sealed (`enc:v1:`) since hub 0.168.0, deployed 2026-10-24 (`hub/internal/store/r975_config_token_seal.go`) |
|
||||
| **added:** app ↔ app (shared Docker daemon) | partly | separate containers and networks; any docker-socket app is root in the guest (R-981) |
|
||||
| **added:** app ↔ controller | partly | apps share a network with the controller; dashboard routes still need a session |
|
||||
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982) |
|
||||
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970) |
|
||||
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971) |
|
||||
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982). **Re-check (2026-10-24, review round 2):** the config bundle's key-only drop-in shipped in agent v0.163.0 (`felhom-agent/configs/felhom-os-apply:230`) |
|
||||
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970). **Re-check (2026-10-24, review round 2):** the config routes are removed and no route takes the hub key — pinned by `felhom-controller/controller/cmd/controller/r970_no_hub_key_door_test.go` (`01` §4 says the same) |
|
||||
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971). **Re-check (2026-10-24, review round 2):** the box checks every rendered compose against its allow-list before any write since controller v0.328.0 (`internal/privpolicy/privpolicy.go`) |
|
||||
| **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`). **Felhom half, 2026-10-23 night:** the last plain tokens sealed in hub 0.168.0 (deployed 2026-10-24 — corrected 2026-10-24; `audits/hub-token-seal-2026-10-23/`); the two plain Longhorn copies of the hub volume already in the folder are R-1243's decision. A household box's version of this attacker (a hacked app sending or mining) is what the upload watch (R-1244, released 2026-10-24 in controller v0.342.0 + hub v0.169.0 — corrected 2026-10-24) detects |
|
||||
|
||||
**[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1281]:** the file manager's one
|
||||
sign-in (R-1100) — FileBrowser trusts its proxy header blindly, and every app container can reach `filebrowser:80`
|
||||
directly; the only guard is that the header NAME is a per-box secret, which traefik's forwardAuth strips from every
|
||||
client request and sets only for a browser holding the dashboard's files pass
|
||||
(`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`, `internal/web/files_sso.go`). The name file's
|
||||
creation race was fixed by review round 2 (controller `b01bb4b`). The pick: FileBrowser joins only the network traefik uses, so no app
|
||||
container reaches it directly. Not built yet.
|
||||
|
||||
## 6. What we accept (do not re-report)
|
||||
|
||||
|
||||
@@ -22,7 +22,7 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42
|
||||
| 2 | controller `POST /api/server/poweroff` (`internal/web/poweroff_handlers.go`) | dashboard login (`RequireAuth` + CSRF), POST only, the word again, the controller's own backup/restore/app-update check, the `host_poweroff` feature gate (MinAgent 0.155.0) |
|
||||
| 3 | agent `POST /host/poweroff` (`internal/localapi/hostpoweroff.go`) | `withGuest` token scope; the guards (§3); the record (`internal/poweroff`) |
|
||||
| 4 | agent → hub | a host report carrying `deliberate_off.requested_at`, waited for until the hub ACCEPTED it (≤ 60 s, `Loop.SetReportedHook`) |
|
||||
| 5 | agent | `pct shutdown <vmid> --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors, `FELHOM_POWEROFF`) |
|
||||
| 5 | agent | `pct shutdown <vmid> --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors: `pct ^shutdown [0-9]+ --timeout [0-9]+$` in `FELHOM_GUESTHOOK`, and `systemctl poweroff` in `FELHOM_POWEROFF` — `felhom-agent/configs/felhom-agent.sudoers:81-90`; corrected 2026-10-24) |
|
||||
| 6 | hub (`store/deliberateoff.go`) | `hosts.deliberate_off_at`; silence alarms held 14 days (§4) |
|
||||
| 7 | the household | presses the machine's power button; the first report of the new boot has no `deliberate_off` → the hub clears it |
|
||||
|
||||
@@ -33,11 +33,11 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42
|
||||
| first 10 min after boot | `too_soon_after_boot` | `TooSoonAfterBoot` |
|
||||
| one accepted request per boot (file keyed by `/proc/stat` btime) | `already_requested_this_boot` | `SecondRequestSameBootRefused`, `poweroff/state_test.go` |
|
||||
| any other guest or VM running (`/proc`: `lxc-start -n N`, `kvm -id N`) | `other_guests_running` | `RefusedWhileOtherGuestsRun`, `RunningGuestsFromProc` |
|
||||
| heavy-operation gate held (backup, restore test, OS step, sweep, trim) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` |
|
||||
| heavy-operation gate held (backup, restore test, OS step, sweep, trim, and since agent v0.164.0 the weekly upload line test — corrected 2026-10-24) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` |
|
||||
| backup job, controller swap, drive format, NAS verify, escrow ceremony, Proxmox vzdump | `backup_running` / `controller_update_running` / `drive_operation_running` / `escrow_ceremony_running` | `RefusedDuringVzdump`, `FailsClosedWithoutTaskReader` |
|
||||
| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` |
|
||||
|
||||
**„A Proxmox vzdump runs” (agent v0.156.1, R-1008):** a vzdump task for the guest in Proxmox's active list whose worker process still lives — Proxmox's own rule (the UPID's pid + pstart against `/proc/<pid>/stat`). After a power cut the dead task stays listed for ~30–60 s; it no longer counts. Doubt (an unparsable task, an unreadable `/proc`) still refuses. The same answer serves the start-up stale-lock pass, the guest-power watchdog and the controller supervisor (`stalelock_inflight_test.go`).
|
||||
| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` |
|
||||
|
||||
**[DESIGN]** A cancelled power-off leaves the box on (gate released) — better a box that did not switch off than a
|
||||
backup, update or format cut in the middle. **[DESIGN]** A failed guest shutdown does not stop the power-off: Proxmox's
|
||||
@@ -86,5 +86,15 @@ alternative (an out-of-band confirmation) needs a second channel the box does no
|
||||
|
||||
## 6. Not done here
|
||||
|
||||
- The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).
|
||||
- ~~The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).~~ Published — `website/gyik.html:128,603` (corrected 2026-10-24).
|
||||
- The BIOS „after power loss" value of the N100 was not read (no monitor); the household text promises only the button.
|
||||
|
||||
## Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
|
||||
|
||||
- **A re-enroll while the household power-off hold is active counts as „gone"** (a reinstall over the old host), not
|
||||
„live" (R-1154; `hub/internal/api/reenroll.go:67-68`).
|
||||
- **Setting and clearing the deliberate-off time each record an info event** — `host_deliberate_off`,
|
||||
`host_deliberate_off_ended` — on the customer's event list; neither is mailed (`hub/internal/api/deliberate_off.go:12-13`).
|
||||
- **The confirmation word is accepted in three forms:** „KIKAPCSOLÁS", the unaccented „KIKAPCSOLAS" (a foreign keyboard)
|
||||
and „TURN OFF" (`felhom-controller/controller/internal/web/poweroff_handlers.go:28-29`).
|
||||
|
||||
@@ -165,7 +165,12 @@ directory) with that tag. Facts the link code needs:
|
||||
**[FACT]** An unlisted writable `${…_PATH}` bind is MANDATORY (every tier, off-site included); a plain host path is in
|
||||
no capture set (`stacks/classify_binds.go` `classifyRoot`). Measured over the live templates: Transmission's unlisted
|
||||
`watch` bind goes off-site; the download folders are `optional`, so a second drive gets a nightly copy of every
|
||||
download (R-1019). **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
|
||||
download (R-1019). *(Fixed since: catalog `c50adfc` marks the download and watch folders `excluded` —
|
||||
`app-catalog-felhom.eu/templates/transmission/.felhom.yml:46-51`; noted 2026-10-24.)* **[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1283]:** the same commit also
|
||||
marks the download folders of NZBGet, SABnzbd and MeTube `excluded` (`templates/{nzbget,sabnzbd,metube}/.felhom.yml`).
|
||||
NZBGet and SABnzbd stay excluded (their files can be fetched again). The pick for MeTube: `optional`, so the
|
||||
second-drive copy carries it and the off-site copy does not — a MeTube video can be gone from its source site. Not
|
||||
built yet. **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
|
||||
media is no single app's data; whether the library gets its own household-level backup is a separate decision.
|
||||
|
||||
## 8. Removal and export assume one drive per app
|
||||
@@ -192,6 +197,13 @@ household's existing files under `media/torrents` stay where they are.
|
||||
Evidence: `audits/media-drive-events-2026-10-14/` (K1–K7). „Idempotent" = a second run changes nothing (K2: run 2 left every entry, folder and library as it
|
||||
was). „Household" = a step only the household can take.
|
||||
|
||||
> *Dated note 2026-10-24 (review round 2): this table is the 2026-10-14 MEASUREMENT („today" = then). As built (§11):
|
||||
> on a drive event the controller only REWRITES the binds; the media apps restart when the household presses
|
||||
> „Újraindítás most" or „Később" (03:30) — the file browser alone recreates itself (`internal/stacks/media_library.go`,
|
||||
> `internal/web/media_events.go`); a safe removal keeps the drive REGISTERED and its return re-binds with no household
|
||||
> step (`SetEjected`, `media_events.go`); and since controller `b01bb4b` (unreleased) neither restart recreates an app
|
||||
> the household stopped (R-721).*
|
||||
|
||||
| event → app | action | measured call | idempotent | household | evidence |
|
||||
|---|---|---|---|---|---|
|
||||
| **added** — every media app | the compose gains the drive's bind → `up -d` recreates it (the compose CHANGED, so today's restart works) | controller compose `up -d` | yes | no | K5-1: 1–10 s per app |
|
||||
@@ -258,3 +270,12 @@ save path when its client is saved before the wiring; **a Plex status probe (`/l
|
||||
unclaimed Plex because Plex answers 401 for a moment while it starts — never gate on a status a starting app also
|
||||
returns** (G7f).
|
||||
|
||||
|
||||
|
||||
**[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1282]:** the „Később"
|
||||
03:30 media restart is an IN-MEMORY timer — a controller restart forgets it and only the notice stays
|
||||
(`internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`). The pick: the controller stores the due
|
||||
time and runs the restart after a restart of its own. Not built yet.
|
||||
|
||||
**[DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]:** the Sonarr/Radarr API key is READ from
|
||||
`/config/config.xml`; the „set at install" option §6 calls cleaner was not built (`internal/mediawire/mediawire.go:336-345`).
|
||||
@@ -15,8 +15,8 @@
|
||||
| Place | Found by | Identified by | Mounted | What lives there | The backup page says |
|
||||
|---|---|---|---|---|---|
|
||||
| Host root / PVE `local` | PVE storage `local` [FACT] A/internal/storage/observe.go:424; never a candidate [FACT] A/internal/storage/role.go:53-79 | `path:` / `store:local` [FACT] A/internal/storage/durableid.go:54-67; role system | host `/`, `/var/lib/vz` | the **local whole-guest copies** (vzdump, retention 3) [FACT] E/scripts/felhom-host-install.sh:730-734; `07` §6.1 | `backup.target.degraded` „…ugyanazon a lemezen van, mint a rendszer…" [FACT] C/internal/web/backup_target_offer.go:78-100 |
|
||||
| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — a Go constant, not i18n [FACT] backup.go:359-361, :1387-1410 |
|
||||
| Enrolled drive (`/mnt/<name>` on the host → `/mnt/felhom-drives/<name>`) | `GET /disks/candidates`: **whole disks only** [FACT] A/internal/storage/candidates.go:33-53 | mount: `uuid:<fs-uuid>`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 |
|
||||
| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — *corrected 2026-10-24:* the i18n keys `note.backup.single_copy` / `note.backup.single_copy_unproven`, picked by the agent's verdict [FACT] C/internal/backup/backup.go:1485-1494 |
|
||||
| Enrolled drive (`/mnt/<name>` on the host → `/mnt/felhom-drives/<name>`) | `GET /disks/candidates`: whole disks, each carrying its nested partition candidates (empty / filesystem / in use; corrected 2026-10-24) [FACT] A/internal/storage/candidates.go:84-90 | mount: `uuid:<fs-uuid>`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 |
|
||||
| Network share | customer input [FACT] A/internal/localapi/netstorage.go:36-40 | none („no durable-id, no SMART") [FACT] netstorage.go:15-19 | `.automount` under `/mnt/felhom-drives/<name>` [FACT] A/internal/storage/netmount.go:18-31 | browse only; never media, never Tier-2 [FACT] media_library.go:83; tier2.go:141-143 | `err.backup.halozati_tarolo_nem_lehet_a_2` [FACT] tier2.go:37-40 |
|
||||
| USB disk | same as an enrolled drive; transport never decides [FACT] C/internal/settings/settings.go:609-614 | same; the bridge may pass the real identity (N100: yes) [FACT] spike case 4a | same | same | same; SMART may be missing → „Nincs adat" |
|
||||
|
||||
@@ -126,7 +126,7 @@ binds both. **Creating or deleting partitions: never from the dashboard in this
|
||||
## 7. The installer
|
||||
|
||||
**[FACT]** Today `step_grows` gives the guest's one data volume (`mp0`) 70 / 130 / 250 GiB by the thin pool's free space
|
||||
(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh:2261-2275). Measured pools: 128 GB
|
||||
(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh `step_grows`, now at :2475 — line corrected 2026-10-24). Measured pools: 128 GB
|
||||
disk → 54 GiB, 200 GB → 113 GiB, 512 GB → 349 GiB (PVE default layout; root capped at 96 GiB).
|
||||
|
||||
**[DESIGN] The rule:** `mp0` = 70 GiB (pool < 300), 130 (300–800), 250 (≥ 800). Reserve = **max(16 GiB, 10 % of the
|
||||
@@ -231,10 +231,25 @@ sizes ruled above in §7):
|
||||
the public installer 1.33.0, agent 0.158.1 and golden 0.318.0 vouched) → data volume 100 GiB, internal place 213 GiB,
|
||||
registered as the default drive with no press; Jellyfin installed with no extra step onto it; a whole-box copy holds nothing
|
||||
of it; partitions formatted and used through the wizard; the honest backup offer on a VM. **Not as written:** an empty part
|
||||
asks for the typed confirmation (the probe reads the parent's partition table — R-1093); a 200 GB disk gets no internal
|
||||
asks for the typed confirmation (the probe reads the parent's partition table — R-1093) *(fixed since in agent v0.160.0:
|
||||
the parent's PTTYPE is set aside when `blkid -p` proves the node is a partition of that scheme — `git tag --contains
|
||||
f7b7308`; noted 2026-10-24)*; a 200 GB disk gets no internal
|
||||
place under the ruled sizes (15 GiB < 20).
|
||||
|
||||
Also in build B: R-1084 (the whole-box backup offer asks the same-disk question) and R-1082 (`/mnt/sys_drive` is never
|
||||
offered to attach, by device). A bundle that adds a path needs a step bundle on boxes with an older wrapper (R16, R-880):
|
||||
0.158.1 was delivered through `0.158.1-step1`.
|
||||
|
||||
## Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
|
||||
|
||||
Read at `file:line` on `main` by the main session:
|
||||
|
||||
- **`systemd-detect-virt --vm` is read once per agent process**; empty output becomes `unknown`, which is not `none`,
|
||||
so every disk then reads unknown/vm — the fail-closed direction (`felhom-agent/internal/storage/identity.go:195-205`).
|
||||
- **A path the controller cannot read counts as the SAME disk** (R-668, v0.269.0): no second copy is ever claimed on
|
||||
it (`felhom-controller/controller/internal/backup/backup.go:1436-1450`).
|
||||
- **The internal place („Belső tárhely") refuses eject, decommission and wipe** — the controller (`refuseInternalLifecycle`)
|
||||
and the agent both — and allows moving its data to another drive (`internal/web/internal_place.go:20-26`).
|
||||
- **The internal place's size subtracts EVERY thin volume already promised in the pool**, not only the 32 GiB rootfs and
|
||||
the data volume (`felhom-agent/configs/felhom-internal-place:11-14`).
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
# `05-hub-architecture.md` — critical review (grounded against felhom-hub v0.6.3 source + Parts 01–04)
|
||||
|
||||
> **HISTORICAL (banner added 2026-10-24, review round 2).** A dated review of hub v0.6.3. Many of its facts have moved
|
||||
> since — for example the stale threshold is 45 min by config (code default 30 min), the operator cooldown is per event
|
||||
> type (`notify/dispatcher.go` `operatorCooldownFor`), and the hub→controller push was retired in v0.26.0. Read
|
||||
> `05-hub-architecture.md` and `08-alarm-ladder.md` for the current behaviour; this file is kept as the record of what
|
||||
> the review found then.
|
||||
|
||||
Method: every claim about the existing hub was checked against `felhom.eu/hub/` source; every
|
||||
cross-doc claim against Parts 01/03/04. Citations are `file:line`. Severity: **blocking** (wrong /
|
||||
breaks an assumption) · **should-fix** (real gap or contradiction, low blast) · **minor**.
|
||||
|
||||
@@ -5,6 +5,21 @@
|
||||
> No matrix is designed here, no target architecture is proposed, and no gap is ranked.
|
||||
>
|
||||
> **This is the input to a redesign, not the redesign.**
|
||||
>
|
||||
> **WHAT HAS MOVED SINCE (banner added 2026-10-24, review round 2 — this inventory records 2026-07-28 and is kept as
|
||||
> it was; each line below was read at `file:line` on `main` that day):**
|
||||
> - The guest has ONE data volume, `mp0` at `/var/lib/felhom`, whose `docker/` and `sys_drive/` folders are bound in the
|
||||
> guest; there is no `mp1` (`felhom-agent/configs/build-golden.sh:31-38,99`) — the „rootfs + mp0 + mp1" lines are history.
|
||||
> - A schema-2 recovery unit (D5) carries the PORTABLE secrets; a restore takes the database password from the unit
|
||||
> (R-1106) — `felhom-controller/controller/internal/backup/recovery_unit.go:24-27`, `restore_dbcreds.go`.
|
||||
> - The second-drive copy's `recovery-unit/` IS read: `RestoreTier2Unit` restores the full unit from the mirror (R-102)
|
||||
> — `internal/backup/tier2_restore.go:197`.
|
||||
> - The off-site reconstitution replays the snapshot's named-volume tars, volumes first (R-354) —
|
||||
> `internal/backup/offbox_reconstitute.go`.
|
||||
> - ep0's per-household prune jobs are keep-weekly 2 at 03:30 (R-1162/R-1165, `scripts/felhom-tenantsync.sh:187-197`);
|
||||
> the local whole-guest tier is keep-daily=3 (R-1149, `felhom-agent/internal/config/config.go:553-559`).
|
||||
> - restic is the pinned official release, checked by version and checksum (R-1105, `felhom-controller/controller/Dockerfile:43-46`),
|
||||
> no longer Debian's 0.14.0.
|
||||
|
||||
## 0. Method, baselines, and what "verified" means here
|
||||
|
||||
|
||||
@@ -25,6 +25,8 @@
|
||||
|
||||
**Gap / roadmap (as written):** Memory-guard FIRING is not live-shown (T-RES-MEMGUARD never fired: ample RAM / auth-walled) — implemented + unit-level only
|
||||
|
||||
**Re-checked 2026-10-24 (catalog sweep, `audits/catalog-sweep-2026-10-24/README.md`):** a fresh box from the public ISO 1.30.0, controller 0.342.0, Tester4. Every catalog app through `POST /api/stacks/<app>/deploy` one at a time, timed to the front door (median 61 s, Cal.com 278 s), then first login, one item, backup, keep-data remove, reinstall, unit restore, remove with data. 62 installed and answered; Wanderer installed but cannot reach its database without DNS (R-739); plant-it and wger are not offered (lifecycle, by design). 52 work, 10 work with a problem (R-1269–R-1277), 0 broken. The memory guard still did not fire (one app at a time) — the gap above stands.
|
||||
|
||||
---
|
||||
|
||||
<a id="b2"></a>
|
||||
@@ -379,3 +381,28 @@
|
||||
**Gap / roadmap (as written):** Remaining ~39 apps are legacy-class by design (unit-only offsite)
|
||||
|
||||
---
|
||||
|
||||
## B23 — An urgent security fix reaches the boxes at night (the security-fix step)
|
||||
|
||||
*Map row: [B23](../00-capability-map.md#b23) · added 2026-10-23.*
|
||||
|
||||
**Scenario:** an app has a published hole (a CVE or GHSA id with its notice) and a fixed upstream version. CC walks the
|
||||
step on scratch 9202 through the product's Update (`box_walk.py --security`: install at FROM, the guarded Update, the
|
||||
four version observables at TO, a 600 s memory soak), and the only writer turns a proven walk into a `security_fix`
|
||||
ladder step. Every box on controller >= 0.341.0 takes it at night with the usual guards (backup first, undo on a
|
||||
failed health check, held / stopped / needs_person / W+5h respected).
|
||||
|
||||
**Components:** catalog `3898593` (`ladder.py check_security`, the two test-record gates, `upgrade-test.py --write-ladder
|
||||
--security`, `box_walk.py --security`); controller v0.341.0 (`legCandidate`).
|
||||
|
||||
**Status:** **PROVEN-LIVE** on 9202 with the drill catalog, 2026-10-23.
|
||||
|
||||
**Evidence:** `audits/security-lane-2026-10-23/` — the negative control (TO `busybox:1.36.1`) `failed` and the writer
|
||||
refused it; Plex 1.43.4.10903 → 1.43.5.11029 `proven`; on 0.340.0 the night leg logged `plex: skipped — not_proven` and
|
||||
a hand press took the step; on 0.341.0-rc.1 the night leg logged `plex: security fix CVE-2026-96652,… taken` and ended
|
||||
`done` after a second-drive copy and an undo copy.
|
||||
|
||||
**Gap:** an app with no box fixture (Plex today, R-1238) is checked only by health, version and the soak — no data is
|
||||
read back. `files_may_change` is a bench measurement and reads `false` when no bench ran (decision 255 item 7).
|
||||
|
||||
---
|
||||
@@ -219,3 +219,25 @@ Tester 1 run controller 0.311.0. An Immich image bump the repair was not proven
|
||||
**Evidence:** `audits/immich-repair-2026-10-14/` — spike A on 9202 (the five candidates), the scripts on real damage
|
||||
(9202), the live cut on demo-hp 9201 (D/). **Not exercised live:** a lost original (the cut lost none); pinned by unit
|
||||
tests and the 9202 run.
|
||||
|
||||
## F13 — A hacked app is noticed (the upload and CPU watch, the weekly line test)
|
||||
|
||||
*Map row: [F13](../00-capability-map.md#f13) · added 2026-10-24.*
|
||||
|
||||
**Scenario:** an app on a household box sends far more than it says it sends, or burns CPU with no transcode to explain
|
||||
it (the Plex2Shell shape). Once a minute the controller reads every app's exact counters and asks the app (qBittorrent,
|
||||
Transmission, Plex); not explained for 15 / 30 minutes → one operator-only alarm, one clear when it stops. Once a week
|
||||
the host's agent measures the upload line with a 10-second iperf3 upload to ep0 inside the tunnel; qBittorrent's default
|
||||
upload limit is half of the tested line, never over the household's own limit.
|
||||
|
||||
**Components:** controller v0.342.0 (`internal/upwatch`, `agentapi.LineTest`), agent v0.164.0 (`internal/linetest`),
|
||||
hub v0.169.0 (four operator-only events; the System page's „Upload line"), ep0 `felhom-iperf3.service`.
|
||||
|
||||
**Status:** **PROVEN-LIVE** — the alarm path on scratch 9202 with `0.342.0-rc.1` (2026-10-23: a 50 Mbit/s flood hidden
|
||||
behind a real seed → one alarm after 15 min, one clear; a CPU burner in Plex → the same; a real seed → none); the line test
|
||||
on the three test boxes (2026-10-24: 94.3 / 94.4 / 92.6 Mbit/s; „server busy" retried); the qBittorrent limit on
|
||||
demo-felhom (5 789 696 B/s = 50 % of 92.6 Mbit/s, three channels agree).
|
||||
|
||||
**Evidence:** `audits/SPIKE-upload-watch-2026-10-23.md` §7; `audits/line-test-2026-10-24/`. **Not exercised live:** an
|
||||
alarm reaching the operator's mailbox from a real box (the 9202 proof had no hub); the family's line on qBittorrent's
|
||||
page on demo-felhom (pinned by a render test). **Gaps:** R-1250, R-1248.
|
||||
@@ -97,10 +97,11 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
|
||||
## Zone 2 — the internet
|
||||
|
||||
- **Cloudflare** (grey): DNS for the customer domains and for felhom.eu; the **tunnels**; the **geo-block** (only Hungarian visitors); flood protection. The hub holds a Cloudflare key per customer zone and checks that it reaches only that customer's own domain.
|
||||
- **ep0** (grey box with an orange stripe, a Hetzner cloud server in Germany): runs the **WireGuard endpoint** and the **off-site backup server (PBS)**. Inside, one datastore with **namespaces**:
|
||||
- **ep0** (grey box with an orange stripe, a Hetzner cloud server in Germany): runs the **WireGuard endpoint**, the **off-site backup server (PBS)** and, since 2026-10-24, a small **line-speed test server** (iperf3) that answers only inside the tunnel — each box measures its upload once a week (R-1247). Inside, one datastore with **namespaces**:
|
||||
- one namespace **per household** (whole-guest copies, each household's own key),
|
||||
with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
|
||||
- the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
|
||||
- outside PBS, a plain folder: the **release copy** (images + agent packages, D33).
|
||||
Prune jobs on ep0: households keep one copy per week for 2 weeks (`keep-weekly 2`, R-1162, 2026-10-19; every new household's job is made with its namespace, R-1165); `operator` keeps 14 daily + 8 weekly.
|
||||
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
|
||||
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
|
||||
@@ -121,8 +122,9 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
|
||||
|
||||
Arrows from DooPlex (dashed, orange → ep0):
|
||||
- **00:20 "DooPlex copy"**: the code (614 MB), Gitea's database and settings, the **password manager**, the nightly secrets export, the **signing keys**. Encrypted on DooPlex with the **DooPlex off-site key**. The key that writes it **cannot delete** old copies. Tested back every Sunday.
|
||||
- **02:30 "Hub database"**: encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
|
||||
- **Left out on purpose:** the container registry (27.7 GB) — "rebuilt from the code".
|
||||
- **02:30, 08:30, 14:30, 20:30 "Hub database"** (every 6 hours since 2026-10-23, D34): encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
|
||||
- **01:40 "Release copy"** (since 2026-10-23, D33): the newest 3 released images of the controller, hub and samba, and the newest 5 agent packages — not encrypted (nothing secret), into a plain folder on ep0 (`/srv/felhom-release-copy`, its own limited user, outside PBS), about 0.7 GB, capped at 3 GB.
|
||||
- **Left out on purpose:** the rest of the container registry (older tags, test builds, non-Felhom images) — "rebuilt from the code".
|
||||
Arrow ep0 → DooPlex (dashed, 05:00): "**ep0-copy** pull: a second copy of ep0 at home. Keeps 8 weekly copies. A daily job removes a deleted customer's copy within 30 days."
|
||||
|
||||
**The test machines** (small, plain tiles near DooPlex, labelled "disposable"):
|
||||
@@ -145,7 +147,7 @@ Title: **"Who tells a box what to do"**
|
||||
## Bottom strip — "One night, hour by hour"
|
||||
|
||||
A horizontal timeline from **00:00 to 09:00**, two rows:
|
||||
- **DooPlex row:** 00:00 database dumps · **00:20 DooPlex copy → ep0** · **02:30 hub database → ep0** · 03:30 ep0 prunes households · 03:45 ep0 prunes `operator` · **05:00 ep0-copy pull → DooPlex** · 07:30 ep0-copy prune · 08:00 deleted-customer clean-up. Sundays: 04:30 restore tests.
|
||||
- **DooPlex row:** 00:00 database dumps · **00:20 DooPlex copy → ep0** · **02:30 hub database → ep0** (and again at 08:30; every 6 h since D34 — corrected 2026-10-24) · 03:30 ep0 prunes households · 03:45 ep0 prunes `operator` · **05:00 ep0-copy pull → DooPlex** · 07:30 ep0-copy prune · 08:00 deleted-customer clean-up. Sundays: 04:30 restore tests.
|
||||
- **Customer box row:** **02:30 Tier 1** · **03:30 Tier 2** · **04:15 Tier 3 off-site** → app updates → **04:30–08:30 whole-guest backup** (local daily, off-site weekly) → Debian fixes → (when approved) Proxmox, Docker, kernel restart. 05:00 the hub checks every box made its backup.
|
||||
- A small note: "Household mails about a kernel restart go out the day before, 09:00–20:00."
|
||||
|
||||
|
||||
@@ -116,7 +116,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
|
||||
<div style="position: absolute; left: 1552px; top: 460px; width: 136px; height: 112px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #5b6270; border-radius: 6px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>iso.felhom.eu</b><br>households download the installer</span></div>
|
||||
<div style="position: absolute; left: 1552px; top: 580px; width: 136px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">web pages</div>
|
||||
<div style="position: absolute; left: 1272px; top: 738px; width: 400px; font-size: 17px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><b style="font-size: 22px">ep0</b> — Hetzner cloud server, Germany</div>
|
||||
<div style="position: absolute; left: 1272px; top: 766px; width: 400px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;">WireGuard endpoint + off-site backup server (PBS). One datastore, split into namespaces:</div>
|
||||
<div style="position: absolute; left: 1272px; top: 766px; width: 400px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;">WireGuard endpoint + off-site backup server (PBS) + a tunnel-only line-speed test (iperf3). One datastore, split into namespaces:</div>
|
||||
<div style="position: absolute; left: 1272px; top: 818px; width: 192px; height: 150px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #0a3d73; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #0a3d73; color: #ffffff; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">G</span> <b>One namespace per household</b><br>whole-guest copies, each household's own key<br><b>keeps one per week, 2 weeks</b></span></div>
|
||||
<div style="position: absolute; left: 1480px; top: 818px; width: 192px; height: 150px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b><code style="font-size: 16px">operator</code> namespace</b><br>groups “hub database” and “DooPlex copy”<br><b>keeps 14 daily + 8 weekly</b></span></div>
|
||||
<div style="position: absolute; left: 1272px; top: 974px; width: 400px; font-size: 15px; line-height: 19px; font-weight: 400; color: #4a5670; text-align: left;"><b>By design:</b> each household holds the key to its own copies; the operator cannot open them without the household’s recovery code.</div>
|
||||
@@ -133,7 +133,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
|
||||
<div style="position: absolute; left: 1720px; top: 706px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">The hub never connects into a box.</div>
|
||||
<div style="position: absolute; left: 1720px; top: 784px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">00:20 DooPlex copy</div>
|
||||
<div style="position: absolute; left: 1720px; top: 824px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">DooPlex off-site key</div>
|
||||
<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">02:30 hub database</div>
|
||||
<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">every 6 h: hub database</div>
|
||||
<div style="position: absolute; left: 1720px; top: 912px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">hub-DB key</div>
|
||||
<div style="position: absolute; left: 1720px; top: 962px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #0a3d73; text-align: center;">05:00 ep0-copy pull</div>
|
||||
<div style="position: absolute; left: 1720px; top: 1002px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #0a3d73; text-align: center;">ciphertext only</div>
|
||||
@@ -191,7 +191,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
|
||||
<div style="position: absolute; left: 374px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
|
||||
<div style="position: absolute; left: 378px; top: 2040px; width: 220px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>00:20</b> DooPlex copy → ep0</div>
|
||||
<div style="position: absolute; left: 894px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
|
||||
<div style="position: absolute; left: 898px; top: 2000px; width: 230px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>02:30</b> hub database → ep0</div>
|
||||
<div style="position: absolute; left: 898px; top: 2000px; width: 230px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>02:30</b> hub DB → ep0, 6-hourly</div>
|
||||
<div style="position: absolute; left: 1134px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
|
||||
<div style="position: absolute; left: 1138px; top: 2040px; width: 236px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>03:30</b> ep0 prunes households</div>
|
||||
<div style="position: absolute; left: 1194px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
# SPIKE — who is using the upload line, and does the app explain it? (2026-10-23 night)
|
||||
|
||||
Operator ruling 2026-10-23 19:22 (`CONTEXT.md`): measure the line; on a sustained high upload attribute it to an app
|
||||
and ask the app; a mismatch is the alarm; the same shape for CPU; qBittorrent gets a default upload limit from the
|
||||
measured line, household changes respected. Brief Part 2.1. Design written from these numbers: §6 here and
|
||||
`architecture/08-alarm-ladder.md` §9 [DESIGN — proposed]. Evidence and tools: `SPIKE-upload-watch-2026-10-23/`.
|
||||
|
||||
Read first: `08-alarm-ladder.md` (R-388, §8.1), `07-backup-architecture.md` (the night chain: W = 02:30 local, the
|
||||
off-site push after it, the proof at 03:30 UTC), `15-media-library.md`, `13-threat-model.md` (attacker j),
|
||||
`audits/plex2shell-felhom-impact-2026-10-23/README.md` (the 1.5 TB Plex upload). Controller: `internal/metrics/`
|
||||
(`container_metrics` — one row per container per minute, 30 days), `internal/monitor/healthcheck.go` (the instant
|
||||
whole-box CPU warning), `internal/mediawire/` (the app clients and their credentials, `09` §3 decision 248).
|
||||
|
||||
Machines: the three test boxes READ ONLY (their stored metrics; their off-site record; tonight a one-second read of
|
||||
the controller's own counter during the real push); scratch 9202 on demo-hp (throwaway qBittorrent, Transmission, Plex,
|
||||
Jellyfin through the product's deploy); a throwaway receiver LXC **9471 `cc-uw-recv`** on demo-hp's scratch storage
|
||||
(never the thin pool). Every flood went 9202 → 9471 on demo-hp's own bridge; nothing went toward the internet.
|
||||
|
||||
## 1. Normal levels (stored metrics, three boxes)
|
||||
|
||||
History: **demo-hp** 2026-09-23 02:00 → 10-23 18:09 UTC (30.7 d, 998 216 rows); **demo-felhom** (the N100) same span
|
||||
(30.7 d, 268 120 rows); **Tester 1** 2026-10-04 19:41 → 10-23 18:08 (18.9 d, 294 433 rows). Per app: upload from the
|
||||
`net_tx_bytes` deltas (one-minute samples), CPU in percent of ONE core. Full tables: `s1-levels-per-app.txt`.
|
||||
|
||||
| box | the busiest uploaders (p50 / p95 / max Mbit/s) | the busiest CPU (p95 / max % of a core) |
|
||||
|---|---|---|
|
||||
| demo-hp | qbittorrent 0.02 / 1.2 / 4.0; immich-server 0.03 / 0.13 / 243 (one minute); felhom-controller 0 / 0.003 / 17.8 (the push) | grimmory 272 / 305 (a scan, 94 samples); immich-ML 32 / 548; romm 3 / 442; paperless 0.3 / 404 |
|
||||
| demo-felhom | immich-server 0.67 / 5.95 / 12.9 (32 samples); jellyfin 0 / 0.85 / 2.9 | immich-ML 295 / 296 (an import); onlyoffice 131 / 148; paperless 137 / 140 |
|
||||
| Tester 1 | qbittorrent 0.007 / 0.13 / 42.4; transmission 0.001 / 0.013 / 57.4; controller 0 / 0.003 / 37.5 | paperless 0.7 / 508; controller 46 / 423; plex 4.3 / 198 |
|
||||
|
||||
**How long a normal high lasts** (`s1-longest-runs.txt` — the longest run of consecutive minutes): upload ≥ 5 Mbit/s:
|
||||
**3 min** at most (qBittorrent, Tester 1); ≥ 1 Mbit/s: 9 min. CPU ≥ 150 %: **7 min** at most (Immich machine learning);
|
||||
≥ 80 %: 13 min (Kimai). Busiest upload hours (UTC): demo-hp 03h, 02h; demo-felhom 16h, 05h; Tester 1 07h, 18h — no fixed
|
||||
pattern beyond the night push. For scale: DooPlex's hacked Plex sent ~474 GB in one day (≈ 44 Mbit/s on average, for
|
||||
hours) and used ~10 cores.
|
||||
|
||||
`felhom-samba` runs with the host network: its counter is always 0 (no own network namespace) — the watch cannot see it.
|
||||
|
||||
## 2. The line without a test
|
||||
|
||||
**What the box already sees:** the off-site push (restic, inside the controller container) is the box's only big
|
||||
upload of its own. At the stored one-minute sample (`s2-controller-push-per-night.txt`): the controller's best five
|
||||
minutes per night exceeded 2 Mbit/s on **2 of 31 nights** on demo-hp (5.1 and 6.9 Mbit/s; best single minute 17.0),
|
||||
**1 of 19** on Tester 1 (7.6; best minute 37.5) and **0 of 31** on demo-felhom. The off-site record keeps the push's
|
||||
duration (33 s on demo-felhom, 1 m 8 s on Tester 1, 3 m 19 s on demo-hp on 2026-10-23) and **no byte count**.
|
||||
|
||||
**Conclusion:** at a one-minute sample the push shows the night's DATA SIZE (a few MB, incremental), not the line. Not
|
||||
enough as it stands. Two changes make it usable: sample once a SECOND during the push (the controller knows when its
|
||||
push runs and can read its own counter), and let a night count only when its best 10 seconds moved ≥ 10 MB. Even then
|
||||
it is a **lower bound** (restic over SFTP to the Storage Box may not fill the line; the family uploads too).
|
||||
|
||||
**Tonight's one-second read of the real push** (2026-10-24 ~02:16Z, three boxes, read only): §2.1 below.
|
||||
|
||||
### 2.1 One-second samples of the real push
|
||||
|
||||
Read 2026-10-24 02:08–02:30Z, once a second, the controller container's own transmit counter (the counter the watch
|
||||
reads: `eth0` in the controller's network namespace), on the three test boxes, read only
|
||||
(`SPIKE-upload-watch-2026-10-23/night-push-1s-<box>.txt`; the reader `tools/nightsample.sh`).
|
||||
|
||||
| Box | Sent in the 22 min | Best 10 s | 1-second peak | Counts (≥ 10 MiB in the best 10 s)? |
|
||||
|---|---|---|---|---|
|
||||
| demo-hp | 19.3 MB | **12.6 MB in 10.1 s = 10.0 Mbit/s** (02:17:32Z) | 70.3 Mbit/s | **yes** → line „at least 10 Mbit/s" |
|
||||
| demo-felhom | 3.3 MB | 3.2 MB = 2.5 Mbit/s (02:15:22Z) | 13.7 Mbit/s | no |
|
||||
| Tester 1 | 3.6 MB | 3.5 MB = 2.7 Mbit/s (02:15:40Z) | 14.7 Mbit/s | no |
|
||||
|
||||
**What it says:** one night in three counted. The push is a burst of a few seconds (3–9 seconds above 100 kB/s), so
|
||||
even the counting night gives a low floor: 10 Mbit/s, while one second of the same push reached 70 Mbit/s. The design
|
||||
keeps the best counting night of the last 14, so the line rises only when a bigger night comes. **Consequence for the
|
||||
qBittorrent limit (R-1245):** 50 % of tonight's floor is ~5 Mbit/s on demo-hp — likely far below its real line. This is
|
||||
the known lower-bound limit (R-1247), now measured; it is written into R-1245 so the operator decides with it.
|
||||
|
||||
## 3. The line with a test — NOT run tonight
|
||||
|
||||
| Candidate | Ours? | Terms | Cost per test | Tonight |
|
||||
|---|---|---|---|---|
|
||||
| ep0 (a receiver on our Hetzner box) | yes | ours | 10 s × line ≈ 125 MB at 100 Mbit/s | **not built** — a change on ep0 is forbidden tonight |
|
||||
| the box's own Storage Box sub-account, through the same transport as the push | yes (Hetzner, unlimited traffic) | Hetzner's — ours | same | not tried — a test file in the account that holds the restic repository needs its own design (the repo check, the quota) |
|
||||
| Cloudflare R2 (where the ISO lives) | yes | ours | same + R2 class-A writes | not used — its write key is the operator's ISO-publish key; a test object in a public bucket is wrong |
|
||||
| public speed tests (speed.cloudflare.com, LibreSpeed servers) | no | **not read** | — | not used (the brief: no third-party test without its terms) |
|
||||
|
||||
A quiet hour: never 02:30–05:30 local (dumps 02:30, second drive, off-site push, the 03:30 UTC proof). The family shares
|
||||
the line, so take the highest of several runs. **Not needed for the first build** — §6 works with the push's lower
|
||||
bound and an absolute floor; an active test is a decide row (R-1247).
|
||||
|
||||
## 4. Can the box ask each app? (measured on 9202 with throwaway installs; a private torrent 9202 → 9471)
|
||||
|
||||
Set-up: qBittorrent 5.2.4 and Transmission 4.1.3 installed through the product (`POST /api/stacks/<n>/deploy`; the
|
||||
first try was refused BEFORE anything was made — the data place must be the drive itself; two empty helper folders
|
||||
removed). The same 2 GiB file (an AES-CTR keystream of a fixed key) on both sides; a private torrent; a 40-line HTTP
|
||||
tracker on 9471 (`tools/tracker.py`); `transmission-daemon` on 9471 as the leecher. The admin passwords were read from
|
||||
the containers at use and never printed.
|
||||
|
||||
| App | Its own number | Measured | Limit | Household's limit visible? |
|
||||
|---|---|---|---|---|
|
||||
| **qBittorrent** | `/api/v2/transfer/info` → `up_info_data` (cumulative), `up_info_speed` | seeding: app **131.1 Mbit/s** vs container **134.6–140.5** → **×1.03–1.07** (`s4-qb-seed-nolimit.jsonl`) | `transfer/setUploadLimit` (bytes/s) = `preferences.up_limit`; stored in KiB/s (2 500 000 → **2 499 584**); survives a restart (`Session\GlobalUPSpeedLimit=2441`); measured: ~20 Mbit/s held (`s4-qb-seed-limit20.jsonl`) | yes: `transfer/uploadLimit` ≠ the value the box stored |
|
||||
| **Transmission** | RPC `session-stats.current-stats.uploadedBytes`, `uploadSpeed` | seeding: app 487.1 / 390.3 / 115.8 vs container 487.6 / 390.7 / 117.9 Mbit/s → **×1.001–1.02** (`s4-tr-seed-nolimit.jsonl`) | `session-set speed-limit-up` (kB/s) + `-enabled` (read back); **its rate under the limit was NOT observed** — the re-added leecher got no connection in 2 minutes | yes: `speed-limit-up-enabled` + value |
|
||||
| **Plex** | `/status/sessions`, `/transcode/sessions` (`MediaContainer.size`) | unclaimed 1.43.5: both 200 without a token from the docker network, `{"size":0}`; a claimed server needs its PlexOnlineToken (read at use from Preferences.xml, as mediawire does — no new secret) | — | — |
|
||||
| **Jellyfin** | `/Sessions` | **401** without a key; the key named `felhom` exists only when the household made it (mediawire's „one household step") — the box cannot create one at install without the admin login it does not have | — | — |
|
||||
| Nextcloud, Immich | **no upload-explaining number found** (not measured further); Immich's job queue needs an API key the box does not hold | — | — | → the plain rule (§6) |
|
||||
|
||||
**The fake hacked app** (code inside qBittorrent's own container, `docker exec … python3` → 9471:5001, Plex2Shell's
|
||||
shape): the container sent **~10 100 Mbit/s** (LAN, one bridge) while qBittorrent reported **0** (`s4-qb-flood.jsonl`).
|
||||
|
||||
**The counter itself:** the Docker Engine API's one-shot stats (`networks.*.tx_bytes`) equal the kernel's counter in
|
||||
the container's network namespace to the byte; `docker stats`' text (what `internal/metrics` stores) has three
|
||||
significant digits — ~10 MB steps at a GB total, ~1.3 Mbit/s at a one-minute sample. The watch reads the Engine API.
|
||||
|
||||
**A limit the hacked app controls:** an attacker with code in the app's container can also answer its API. The check
|
||||
catches an attacker who does not know about it; one who does could lie. §5 is the answer to that.
|
||||
|
||||
## 5. A universal upload ceiling — spike only
|
||||
|
||||
A helper container sharing the app's network namespace (`docker run --rm --network container:qbittorrent --cap-add
|
||||
NET_ADMIN alpine:3.20` + `iproute2-tc`) added `tc qdisc … root tbf rate 20mbit burst 64kb latency 400ms` on the app's
|
||||
`eth0` **inside the unprivileged 9202 guest**. The same flood then ran at **19.9–20.1 Mbit/s** (`s4-qb-flood-tbf.jsonl`),
|
||||
qBittorrent's API kept answering, and `tc qdisc del` restored it. Costs: a helper image with `tc` (a new external image,
|
||||
or `tc` shipped in the controller image), `NET_ADMIN` for that helper, re-applied after every app restart (the qdisc
|
||||
dies with the namespace), upload only. **Not built** (brief). It would cap a hacked app's flood whatever its API says.
|
||||
|
||||
## 6. The design (numbers from §1–§5; full text `08-alarm-ladder.md` §9)
|
||||
|
||||
1. **Line:** one-second samples of the controller's own sent bytes during the off-site push; the best ≥ 10-second rate
|
||||
per night; a night counts when that window moved ≥ 10 MB; the line = the highest counting night of the last 14;
|
||||
settings.json `upwatch.line`; shown on the system settings page as „legalább N Mbit/s" („még mérjük" until known).
|
||||
2. **Trigger:** an app (its containers summed, exact counters, once a minute) sends ≥ **max(5 Mbit/s, 25 % of the line)**
|
||||
for **15 minutes** — five times the longest normal run at ≥ 5 Mbit/s (3 min). Per app, not per box: the box cannot
|
||||
tell LAN from internet bytes inside its controller container, and the per-app view is what the check compares.
|
||||
3. **Attribution:** the app's compose project; infrastructure (controller, traefik, cloudflared, Samba) never judged.
|
||||
4. **Explanation:** qBittorrent / Transmission: explained when container bytes ≤ 1.25 × the app's own count + 1 Mbit/s
|
||||
(measured overhead 1–7 %); Plex: any viewer explains the upload (session bandwidth not measured — the conservative
|
||||
half; no viewer + a big upload = the Plex2Shell signature). Not explained → **`app_unexplained_upload`**,
|
||||
operator-only, one per episode, a clear after 15 quiet minutes. No explainer: alarm only at ≥ 10 × its own p95 for
|
||||
60 minutes, after ~20 h of history, worded „no app number to compare".
|
||||
5. **CPU:** ≥ **max(150 %, 50 % × cores)** of one core for **30 minutes** (four times the longest normal run at ≥ 150 %),
|
||||
explained by a Plex transcode or the box's own work on the app (backup, update, deploy) → else
|
||||
**`app_unexplained_cpu`**. No explainer: ≥ max(200 %, 75 % × cores) for 60 minutes.
|
||||
6. **qBittorrent limit:** 50 % of the line (CC's pick — R-1245), every 15 minutes, re-set when the line moves > 20 %;
|
||||
never over a household's limit (a limit there first, or any change from the stored value). No line → nothing set.
|
||||
The household's line (HU + EN in the report).
|
||||
7. **Edge cases:** the night push is the controller's (never judged); the first day: apps with an explainer are judged
|
||||
on the floor (a mismatch is evidence whatever the line), apps without one are not judged before ~20 h of history; a
|
||||
slow or changing line (a Wi-Fi laptop): the line is a max over 14 nights — a falling line lowers nothing until it
|
||||
rolls out (R-1247 notes the active test); two apps at once: each judged on its own bytes; the household's own devices:
|
||||
invisible to the box — the trigger uses the box's own bytes only; an app reinstalled from scratch reads as „the
|
||||
household removed the limit" and is left alone.
|
||||
|
||||
## 7. Live proofs on 9202 (controller `0.342.0-rc.1`, commit `a9c1997`, CI run 2270 success)
|
||||
|
||||
Set by hand in 9202's `/etc/felhom-controller-image` (an `-rc` tag is never offered to a household); 9202's hub link
|
||||
is off by design, so every event was handed to the notifier and DROPPED there — the hub side (allow-list, operator-only,
|
||||
the mail line) is proven by hub 0.169.0's tests. 9202's off-site is off, so one line record (40 Mbit/s) was seeded by
|
||||
hand into its settings.json. Logs: `SPIKE-upload-watch-2026-10-23/live/2-9202-watch-log-full.txt`.
|
||||
|
||||
| # | What | Result (UTC) |
|
||||
|---|---|---|
|
||||
| 1 | qBittorrent unlimited, line 40 Mbit/s → the box sets the limit | **19:47:42** „qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)" — read back from qBittorrent: 2 499 584 |
|
||||
| 2 | **a real torrent seeding at the limit** (5 GB, private, 9202 → 9471) | **20:04:42** „qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s)"; no alarm (counter ratio 1.04) |
|
||||
| 3 | **a CPU burner in Plex with no transcode** (4 busy loops, 400 % on a 7-core guest; trigger 350 %) | **20:07:42** „plex used 399% CPU … for 30 min with no work that explains it — operator alarm" → `app_unexplained_cpu`; ONE |
|
||||
| 4 | **the fake hacked app**: a 50 Mbit/s flood from inside qBittorrent's container to 9471, hidden behind the real seed | **20:19:42** „qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s) — operator alarm" → `app_unexplained_upload`; ONE, none more while it ran |
|
||||
| 5 | the burner stopped 20:11 | **20:25:41** „plex: the unexplained CPU stopped … — cleared" → `app_unexplained_cpu_cleared`; ONE |
|
||||
| 6 | the household changes qBittorrent's limit by hand (1 500 000 → stored 1 499 136) at 20:21 | **20:32:42** „qBittorrent's upload limit is the household's — the box leaves it alone from now on"; read back at 20:43: still 1 499 136 |
|
||||
| 7 | the flood stopped 20:26 (the seed went on, explained) | **20:41:41** „qbittorrent: the unexplained upload stopped … — cleared" → `app_unexplained_upload_cleared`; ONE |
|
||||
| 8 | the system settings page (`GET /settings`) | „Feltöltési sebesség: legalább 40 Mbit/s (az éjszakai távoli mentésből mérve)"; qBittorrent's page carried no limit line after step 6 (the household's) — the box-owned branch is pinned by `TestUpwatch_QBLimitLineOnQBittorrentsPageOnly`, not captured live |
|
||||
|
||||
**Found live:** the very first minute after a start failed to reach the apps by name (the bootstrap joins the controller
|
||||
to the app network a few seconds after the start); every later minute answered. That first minute only sets baselines —
|
||||
**NOT-A-FINDING**. The media wiring's generated `docker-compose.override.yml` stays beside the template after a removal
|
||||
(Jellyfin tonight; Plex's from the morning session) — rewritten before every start, harmless — **NOT-A-FINDING**.
|
||||
|
||||
**Teardown, three layers:** machine — the four throwaway apps removed through the product (containers and volumes gone,
|
||||
verified), my test files and probes deleted, the `alpine:3.20` helper image removed, 9202 back on 0.341.0 with the seeded
|
||||
record removed, the rc image removed; host — the receiver LXC 9471 destroyed (`--purge`, its disk gone); hub — none (9202
|
||||
is not on the hub). The night sampler on the three test boxes is reads only.
|
||||
@@ -0,0 +1,15 @@
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.342.0-rc.1
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0-rc.1 Up 6 minutes (healthy)
|
||||
filebrowser gtstef/filebrowser:1.5.6-stable Up 4 hours (healthy)
|
||||
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 4 hours (healthy)
|
||||
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 4 hours (healthy)
|
||||
immich-redis redis:7-alpine Up 4 hours (healthy)
|
||||
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 4 hours (healthy)
|
||||
jellyfin jellyfin/jellyfin:10.11.11 Up About an hour (healthy)
|
||||
paperless-postgres postgres:18-alpine Up 4 hours (healthy)
|
||||
paperless-redis redis:7-alpine Up 4 hours (healthy)
|
||||
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 4 hours (healthy)
|
||||
plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 Up About an hour (healthy)
|
||||
qbittorrent qbittorrentofficial/qbittorrent-nox:5.2.4-1 Up About an hour (healthy)
|
||||
traefik traefik:v3.7.13 Up 4 hours
|
||||
transmission lscr.io/linuxserver/transmission:version-4.1.3-r0 Up About an hour (healthy)
|
||||
@@ -0,0 +1,10 @@
|
||||
VMID Status Lock Name
|
||||
9201 running demo-hp
|
||||
9202 running demo-hp-scratch
|
||||
9401 stopped upgrade-harness
|
||||
9471 running cc-uw-recv
|
||||
description: CC throwaway 2026-10-23 night%3A upload-watch spike receiver/leecher. Remove after the session.%0A
|
||||
hostname: cc-uw-recv
|
||||
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:04:68:80,ip=dhcp,type=veth
|
||||
rootfs: nvme-scratch:9471/vm-9471-disk-0.raw,size=8G
|
||||
tags: cc;scratch;uw-spike
|
||||
@@ -0,0 +1,9 @@
|
||||
2026/10/23 19:32:41 upwatch_wiring.go:97: [INFO] [upwatch] upload and CPU watch started (line at least 40 Mbit/s, qBittorrent share 50 %)
|
||||
2026/10/23 19:32:41 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] transmission: the app did not answer (Post "http://transmission:9091/transmission/rpc": dial tcp: lookup transmission on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||
2026/10/23 19:32:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_definition_refused (severity warning) — further app_definition_refused events are logged at DEBUG only
|
||||
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] qbittorrent: the app did not answer (Post "http://qbittorrent:8080/api/v2/auth/login": dial tcp: lookup qbittorrent on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||
2026/10/23 19:47:42 service.go:108: [INFO] [upwatch] qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)
|
||||
2026/10/23 20:04:42 watch.go:303: [INFO] [upwatch] qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)); normal use
|
||||
2026/10/23 20:07:42 watch.go:343: [WARN] [upwatch] plex used 399% CPU (of one core; 7 cores) for 30 min with no work that explains it (plex transcodes) — operator alarm
|
||||
2026/10/23 20:07:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu (severity warning) — further app_unexplained_cpu events are logged at DEBUG only
|
||||
@@ -0,0 +1,16 @@
|
||||
2026/10/23 19:32:41 upwatch_wiring.go:97: [INFO] [upwatch] upload and CPU watch started (line at least 40 Mbit/s, qBittorrent share 50 %)
|
||||
2026/10/23 19:32:41 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] transmission: the app did not answer (Post "http://transmission:9091/transmission/rpc": dial tcp: lookup transmission on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||
2026/10/23 19:32:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_definition_refused (severity warning) — further app_definition_refused events are logged at DEBUG only
|
||||
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] qbittorrent: the app did not answer (Post "http://qbittorrent:8080/api/v2/auth/login": dial tcp: lookup qbittorrent on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||
2026/10/23 19:47:42 service.go:108: [INFO] [upwatch] qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)
|
||||
2026/10/23 20:04:42 watch.go:303: [INFO] [upwatch] qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)); normal use
|
||||
2026/10/23 20:07:42 watch.go:343: [WARN] [upwatch] plex used 399% CPU (of one core; 7 cores) for 30 min with no work that explains it (plex transcodes) — operator alarm
|
||||
2026/10/23 20:07:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu (severity warning) — further app_unexplained_cpu events are logged at DEBUG only
|
||||
2026/10/23 20:19:42 watch.go:307: [WARN] [upwatch] qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)) — operator alarm
|
||||
2026/10/23 20:19:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_upload (severity warning) — further app_unexplained_upload events are logged at DEBUG only
|
||||
2026/10/23 20:25:41 watch.go:378: [INFO] [upwatch] plex: the unexplained CPU stopped (15 min below the trigger or explained) — cleared
|
||||
2026/10/23 20:25:41 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu_cleared (severity info) — further app_unexplained_cpu_cleared events are logged at DEBUG only
|
||||
2026/10/23 20:32:42 service.go:111: [INFO] [upwatch] qBittorrent's upload limit is the household's — the box leaves it alone from now on
|
||||
2026/10/23 20:41:41 watch.go:370: [INFO] [upwatch] qbittorrent: the unexplained upload stopped (15 min below the trigger or explained) — cleared
|
||||
2026/10/23 20:41:41 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_upload_cleared (severity info) — further app_unexplained_upload_cleared events are logged at DEBUG only
|
||||
@@ -0,0 +1,2 @@
|
||||
/settings 56743 data-uplink> <span class="settings-label">Feltöltési sebesség</span> <span class="settings-value">legalább <span class="mono">40</span> Mbit/s (az éjszakai távoli mentésből mérve)</span> </div> | qb line: none
|
||||
/apps/qbittorrent 49281 no uplink row | qb line: none
|
||||
@@ -0,0 +1,4 @@
|
||||
version v5.2.4 webapi 2.15.1
|
||||
transfer/info {'connection_status': 'connected', 'dht_nodes': 375, 'dl_info_data': 0, 'dl_info_speed': 0, 'dl_rate_limit': 0, 'last_external_address_v4': '37.191.56.193', 'last_external_address_v6': '', 'up_info_data': 5566595787, 'up_info_speed': 0, 'up_rate_limit': 1499136}
|
||||
prefs {'up_limit': 1499136, 'dl_limit': 0, 'alt_up_limit': 10240, 'alt_dl_limit': 10240, 'scheduler_enabled': False, 'limit_tcp_overhead': False, 'limit_lan_peers': True, 'limit_utp_rate': True, 'max_uploads': 20, 'enable_upload_suggestions': False}
|
||||
uploadLimit 1499136 speedLimitsMode 0
|
||||
@@ -0,0 +1,186 @@
|
||||
|
||||
=== demo-hp: 2026-09-23 02:00 -> 2026-10-23 18:09 UTC (30.7 d), 998216 rows
|
||||
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
|
||||
qbittorrent 18 0.019 1.199 4.01 | 0.4 5.4 11.3 |
|
||||
metube 3 0.146 0.146 0.15 | 0.0 0.1 0.1 |
|
||||
immich-server 318 0.031 0.133 242.93 | 0.1 5.6 138.1 | resets=7
|
||||
grimmory-db 102 0.000 0.116 0.35 | 0.0 16.9 102.5 | resets=12
|
||||
immich-postgres 320 0.000 0.066 2.20 | 0.0 8.7 70.2 | resets=7
|
||||
grimmory 94 0.001 0.047 0.77 | 1.1 272.0 305.0 | resets=9
|
||||
immich-redis 320 0.013 0.029 0.18 | 0.6 4.3 5.1 | resets=7
|
||||
romm 26492 0.019 0.027 0.20 | 0.7 3.2 442.4 | resets=97
|
||||
docmost 43499 0.013 0.027 0.23 | 0.1 0.8 158.1 | resets=147
|
||||
rallly 71 0.000 0.017 0.12 | 0.0 8.5 152.1 | resets=10
|
||||
radarr 16 0.002 0.015 0.03 | 0.2 7.2 11.8 |
|
||||
romm-redis 26523 0.005 0.013 0.10 | 0.7 4.4 61.5 | resets=101
|
||||
paperless-webserver 43488 0.005 0.013 0.04 | 0.1 0.3 404.2 | resets=144
|
||||
nextcloud-db 663 0.001 0.013 1.74 | 0.0 6.1 71.8 | resets=11
|
||||
docmost-redis 43532 0.005 0.013 0.13 | 0.7 4.4 6.6 | resets=147
|
||||
cloudflared 43604 0.000 0.013 1.21 | 0.3 0.5 13.8 | resets=6
|
||||
rallly-postgres 71 0.000 0.007 0.01 | 0.0 5.0 8.8 | resets=7
|
||||
prowlarr 5 0.002 0.004 0.00 | 0.1 0.2 0.2 |
|
||||
sonarr 18 0.000 0.004 0.01 | 2.5 8.2 8.8 |
|
||||
felhom-controller 43608 0.000 0.003 17.79 | 1.0 57.9 251.4 | resets=133
|
||||
paperless-redis 43521 0.001 0.003 0.02 | 0.6 4.3 6.1 | resets=148
|
||||
bentopdf 43567 0.000 0.003 0.01 | 0.0 0.0 5.2 | resets=58
|
||||
homepage 78 0.000 0.002 0.01 | 0.0 4.2 4.5 | resets=1
|
||||
bookstack-db 43565 0.001 0.001 0.01 | 0.0 6.1 102.8 | resets=147
|
||||
romm-db 26526 0.000 0.001 0.19 | 0.0 6.0 105.3 | resets=89
|
||||
plex 11974 0.001 0.001 1.10 | 0.3 0.6 124.6 | resets=46
|
||||
nextcloud 662 0.000 0.001 0.80 | 0.0 5.7 150.0 | resets=11
|
||||
immich-machine-learning 320 0.000 0.001 0.39 | 0.2 32.0 548.0 | resets=7
|
||||
docmost-postgres 43533 0.000 0.001 0.02 | 0.0 4.7 29.6 | resets=147
|
||||
bookstack 43546 0.000 0.001 0.01 | 0.0 0.0 63.0 | resets=148
|
||||
adventurelog-postgres 43567 0.000 0.001 1.35 | 0.0 4.7 37.9 | resets=138
|
||||
adventurelog 43549 0.000 0.001 2.16 | 0.0 0.7 204.3 | resets=146
|
||||
ghost 20 0.000 0.001 0.00 | 0.0 6.8 132.9 | resets=2
|
||||
glance 20 0.000 0.001 0.00 | 0.0 0.0 0.0 | resets=2
|
||||
privatebin 43518 0.000 0.001 0.00 | 0.1 0.1 8.1 | resets=140
|
||||
paperless-postgres 43520 0.000 0.001 0.05 | 0.0 4.7 19.1 | resets=148
|
||||
kimai 30136 0.000 0.000 0.00 | 0.0 0.0 107.2 | resets=102
|
||||
home-assistant 14403 0.000 0.000 1.00 | 0.0 0.5 106.9 | resets=62
|
||||
kimai-db 30150 0.000 0.000 0.05 | 0.0 5.9 100.8 | resets=96
|
||||
adventurelog-frontend 43559 0.000 0.000 0.21 | 0.0 3.8 22.8 | resets=126
|
||||
vaultwarden 329 0.000 0.000 0.00 | 0.0 0.1 4.0 | resets=7
|
||||
calibre-web 43547 0.000 0.000 0.01 | 0.0 0.3 127.1 | resets=145
|
||||
opengist 43530 0.000 0.000 0.01 | 0.0 0.0 9.8 | resets=140
|
||||
jellyfin 297 0.000 0.000 5.87 | 0.0 4.4 160.5 | resets=7
|
||||
vikunja 8 0.000 0.000 0.00 | 0.0 0.0 0.0 | resets=2
|
||||
a1-echo 113 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
upbeat_perlman 1 0.000 0.000 0.00 | 31.2 31.2 31.2 |
|
||||
traefik 43608 0.000 0.000 1.21 | 0.1 3.5 92.1 | resets=11
|
||||
practical_johnson 1 0.000 0.000 0.00 | 9.9 9.9 9.9 |
|
||||
peaceful_ptolemy 1 0.000 0.000 0.00 | 14.1 14.1 14.1 |
|
||||
nextcloud-redis 663 0.000 0.000 0.03 | 0.6 4.1 7.9 | resets=5
|
||||
friendly_wilbur 1 0.000 0.000 0.00 | 59.0 59.0 59.0 |
|
||||
filebrowser 43607 0.000 0.000 0.74 | 0.0 0.0 35.1 | resets=17
|
||||
felhom-samba 10 0.000 0.000 0.00 | 0.0 13.5 17.6 | tx counter always 0 (host network?)
|
||||
determined_satoshi 1 0.000 0.000 0.00 | 17.4 17.4 17.4 |
|
||||
busiest UTC hours by bytes sent: 03h 14%, 02h 7%, 13h 5%, 00h 4%
|
||||
|
||||
=== demo-felhom: 2026-09-23 02:00 -> 2026-10-23 18:08 UTC (30.7 d), 268120 rows
|
||||
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
|
||||
immich-server 32 0.667 5.953 12.90 | 0.6 120.1 130.8 |
|
||||
jellyfin 16 0.000 0.854 2.85 | 0.0 72.7 290.7 |
|
||||
immich-postgres 32 0.000 0.766 2.24 | 0.3 8.5 197.6 |
|
||||
romm 16 0.020 0.339 1.07 | 0.5 25.5 100.5 |
|
||||
dawarich 20 0.000 0.313 3.01 | 0.1 5.3 104.4 |
|
||||
immich-machine-learning 32 0.003 0.243 0.42 | 0.2 294.7 295.7 |
|
||||
immich-redis 32 0.017 0.144 0.16 | 0.5 3.9 4.2 |
|
||||
nextcloud-db 16 0.000 0.137 0.43 | 0.0 4.9 5.8 |
|
||||
nextcloud 16 0.000 0.099 0.33 | 0.0 21.7 86.7 |
|
||||
paperless-webserver 21 0.005 0.073 0.08 | 0.0 137.4 139.7 |
|
||||
paperless-postgres 22 0.000 0.048 0.06 | 0.0 4.6 5.0 |
|
||||
kimai-db 19 0.000 0.023 0.15 | 0.0 8.4 38.8 |
|
||||
dawarich-sidekiq 19 0.020 0.023 0.02 | 0.1 0.2 0.2 |
|
||||
claper 20 0.019 0.020 0.02 | 0.5 0.7 0.9 |
|
||||
docmost 22 0.020 0.020 0.02 | 0.0 0.2 0.3 |
|
||||
wanderer 19 0.002 0.015 0.02 | 0.0 0.0 0.0 |
|
||||
cloudflared 43863 0.000 0.013 1.33 | 0.2 0.3 8.3 | resets=11
|
||||
adventurelog 22 0.000 0.013 3.24 | 0.0 0.1 65.8 |
|
||||
grafana 20 0.000 0.012 0.12 | 1.0 1.9 17.3 |
|
||||
wanderer-db 19 0.007 0.011 0.01 | 0.0 3.7 3.7 |
|
||||
dawarich-redis 20 0.009 0.011 0.01 | 0.4 3.7 3.8 |
|
||||
claper-postgres 20 0.011 0.011 0.01 | 0.2 0.5 4.4 |
|
||||
outline 19 0.008 0.008 0.01 | 0.0 7.4 7.7 |
|
||||
nextcloud-redis 16 0.000 0.008 0.02 | 0.4 0.7 1.4 |
|
||||
romm-redis 16 0.006 0.007 0.01 | 0.5 3.7 3.8 |
|
||||
docmost-redis 23 0.005 0.005 0.07 | 0.4 0.7 3.9 |
|
||||
dawarich-db 20 0.001 0.005 0.03 | 0.0 0.1 0.1 |
|
||||
bookstack-db 20 0.001 0.005 0.04 | 0.0 4.7 4.8 |
|
||||
grocy 19 0.000 0.005 0.01 | 0.0 0.0 0.0 |
|
||||
bookstack 20 0.000 0.005 0.04 | 0.0 7.9 47.2 |
|
||||
zipline 20 0.003 0.003 0.00 | 0.0 0.0 0.0 |
|
||||
paperless-redis 22 0.002 0.003 0.00 | 0.4 0.4 0.4 |
|
||||
outline-redis 19 0.003 0.003 0.00 | 0.4 0.4 0.5 |
|
||||
kimai 19 0.000 0.003 0.02 | 0.0 2.4 24.3 |
|
||||
zipline-postgres 20 0.003 0.003 0.00 | 0.0 0.0 0.0 |
|
||||
qbittorrent 11759 0.001 0.003 0.01 | 0.1 0.2 97.5 | resets=25
|
||||
felhom-controller 43863 0.000 0.003 1.73 | 0.1 15.5 122.0 | resets=92
|
||||
bentopdf 18 0.000 0.002 0.00 | 0.0 0.0 0.0 |
|
||||
emby 19 0.001 0.002 0.01 | 0.0 0.1 0.1 |
|
||||
homepage 20 0.000 0.002 0.00 | 0.0 0.0 0.0 |
|
||||
sonarr 11756 0.000 0.001 0.01 | 0.1 0.3 74.7 | resets=25
|
||||
grimmory-db 24 0.000 0.001 0.35 | 0.0 5.0 100.2 |
|
||||
grimmory 23 0.001 0.001 0.00 | 5.7 25.3 35.0 |
|
||||
lubelogger 19 0.000 0.001 0.00 | 0.0 0.2 0.6 |
|
||||
radarr 11756 0.000 0.001 0.17 | 0.0 0.6 79.9 | resets=25
|
||||
glance 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
metube 20 0.000 0.001 0.00 | 0.0 1.3 26.1 |
|
||||
actualbudget 18 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
privatebin 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
plex 18 0.000 0.001 0.00 | 0.7 0.8 0.8 |
|
||||
papra 20 0.000 0.001 0.00 | 0.0 6.8 7.5 |
|
||||
home-assistant 16 0.000 0.001 0.00 | 0.0 19.8 79.0 |
|
||||
wanderer-search 19 0.000 0.001 0.00 | 0.1 0.1 0.1 |
|
||||
karakeep 19 0.001 0.001 0.00 | 1.8 2.0 2.0 |
|
||||
romm-db 16 0.001 0.001 0.00 | 0.0 4.6 4.7 |
|
||||
radicale 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
gramps-web 20 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
tandoor 19 0.000 0.001 0.00 | 0.1 4.7 4.8 |
|
||||
karakeep-chrome 19 0.001 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
outline-postgres 19 0.000 0.001 0.00 | 0.0 0.4 0.7 |
|
||||
seerr 18 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
n8n 22 0.000 0.000 0.00 | 0.8 1.0 110.6 |
|
||||
termix 20 0.000 0.000 0.00 | 0.0 0.4 2.3 |
|
||||
tandoor-postgres 19 0.000 0.000 0.00 | 0.4 1.5 4.1 |
|
||||
audiobookshelf 19 0.000 0.000 0.00 | 0.0 0.0 0.1 |
|
||||
adventurelog-postgres 22 0.000 0.000 0.02 | 0.0 3.8 33.6 |
|
||||
ghost 21 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
transmission 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||
komga 19 0.000 0.000 0.00 | 0.3 6.8 10.6 |
|
||||
calcom 18 0.000 0.000 0.00 | 0.0 3.4 22.3 |
|
||||
vaultwarden 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
sparkyfitness 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
homebox 19 0.000 0.000 0.00 | 0.1 0.2 0.2 |
|
||||
docmost-postgres 23 0.000 0.000 0.01 | 0.0 3.8 73.9 |
|
||||
vikunja 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
adventurelog-frontend 22 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
rallly 20 0.000 0.000 0.00 | 0.0 5.6 110.3 |
|
||||
calibre-web 17 0.000 0.000 0.00 | 0.0 21.3 106.4 |
|
||||
wishlist 20 0.000 0.000 0.00 | 0.0 6.3 125.1 |
|
||||
prowlarr 11738 0.000 0.000 0.00 | 0.0 0.1 77.0 | resets=26
|
||||
opengist 43862 0.000 0.000 0.01 | 0.0 0.0 6.0 | resets=107
|
||||
onlyoffice 22 0.000 0.000 0.00 | 0.0 130.5 147.5 |
|
||||
sabnzbd 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
gitea 19 0.000 0.000 0.00 | 0.0 0.4 4.0 |
|
||||
navidrome 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||
code-server 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
uptime-kuma 19 0.000 0.000 0.00 | 0.7 0.7 0.7 |
|
||||
nzbget 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
gokapi 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
recipe-importer 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
crafty-controller 19 0.000 0.000 0.00 | 0.1 8.3 8.5 |
|
||||
mealie 20 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||
rallly-postgres 20 0.000 0.000 0.00 | 0.0 3.7 3.8 |
|
||||
traefik 43863 0.000 0.000 0.96 | 0.0 3.6 17.9 | resets=13
|
||||
sparkyfitness-server 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||
sparkyfitness-db 19 0.000 0.000 0.00 | 0.0 3.8 3.8 |
|
||||
karakeep-meilisearch 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||
filebrowser 43863 0.000 0.000 0.74 | 0.0 0.0 15.7 | resets=16
|
||||
calcom-postgres 18 0.000 0.000 0.00 | 0.0 3.9 4.0 |
|
||||
busiest UTC hours by bytes sent: 16h 16%, 05h 12%, 09h 6%, 02h 5%
|
||||
|
||||
=== tester-1: 2026-10-04 19:41 -> 2026-10-23 18:08 UTC (18.9 d), 294433 rows
|
||||
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
|
||||
qbittorrent 3857 0.007 0.133 42.40 | 0.1 2.6 92.3 | resets=20
|
||||
transmission 3857 0.001 0.013 57.43 | 0.1 1.6 82.5 | resets=19
|
||||
paperless-webserver 26710 0.005 0.013 0.09 | 0.1 0.7 508.1 | resets=63
|
||||
cloudflared 26751 0.000 0.013 1.63 | 0.4 1.2 29.4 | resets=6
|
||||
prowlarr 3725 0.000 0.012 0.26 | 0.1 0.3 101.8 | resets=8
|
||||
sonarr 3855 0.001 0.004 1.21 | 0.1 4.5 131.6 | resets=20
|
||||
radarr 3855 0.001 0.003 1.52 | 0.1 4.3 103.3 | resets=19
|
||||
paperless-redis 26740 0.001 0.003 0.01 | 0.7 4.2 27.5 | resets=68
|
||||
felhom-controller 26751 0.000 0.003 37.47 | 0.5 46.4 422.6 | resets=62
|
||||
plex 3739 0.001 0.002 1.07 | 0.3 4.3 198.4 | resets=17
|
||||
bookstack-db 26729 0.001 0.001 0.03 | 0.0 5.7 130.6 | resets=66
|
||||
plex-k6 514 0.001 0.001 0.01 | 0.4 5.4 7.2 |
|
||||
bookstack 26722 0.000 0.001 0.04 | 0.0 0.1 72.5 | resets=65
|
||||
privatebin 26744 0.000 0.001 0.00 | 0.0 0.1 14.7 | resets=64
|
||||
paperless-postgres 26740 0.000 0.001 0.02 | 0.0 4.4 65.8 | resets=68
|
||||
traefik 26751 0.000 0.000 1.56 | 0.0 3.4 100.6 | resets=3
|
||||
jellyfin 3648 0.000 0.000 2.83 | 0.0 0.2 181.5 | resets=19
|
||||
xfp-echo 1 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
vaultwarden 1 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
filebrowser 26743 0.000 0.000 0.01 | 0.0 0.0 80.1 | resets=22
|
||||
busiest UTC hours by bytes sent: 07h 13%, 18h 11%, 02h 9%, 17h 9%
|
||||
@@ -0,0 +1,22 @@
|
||||
== demo-hp
|
||||
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
|
||||
kimai 13 0 (-) 0 0 (-)
|
||||
paperless-webserver 6 1 (09-23 04:56) 0 0 (-)
|
||||
immich-machine-learning 4 4 (10-14 00:54) 0 0 (-)
|
||||
romm 3 2 (10-23 15:36) 0 0 (-)
|
||||
adventurelog 2 1 (09-30 11:29) 3 0 (-)
|
||||
adventurelog-postgres 0 0 (-) 3 0 (-)
|
||||
felhom-controller 3 1 (09-23 11:37) 2 2 (09-28 13:44)
|
||||
immich-server 2 0 (-) 3 2 (10-14 00:53)
|
||||
== demo-felhom
|
||||
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
|
||||
immich-server 2 0 (-) 8 1 (10-13 05:32)
|
||||
immich-machine-learning 7 7 (10-13 16:39) 0 0 (-)
|
||||
paperless-webserver 6 0 (-) 0 0 (-)
|
||||
onlyoffice 4 0 (-) 0 0 (-)
|
||||
== tester-1
|
||||
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
|
||||
plex 10 5 (10-14 17:20) 1 0 (-)
|
||||
qbittorrent 2 0 (-) 9 3 (10-14 18:08)
|
||||
sonarr 9 0 (-) 2 0 (-)
|
||||
transmission 1 0 (-) 6 2 (10-16 07:17)
|
||||
@@ -0,0 +1,90 @@
|
||||
|
||||
== demo-hp: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
|
||||
2026-09-23 max1= 3.14 best5= 0.91 busy_min= 0 peak_at=02:17Z
|
||||
2026-09-24 max1= 3.04 best5= 0.73 busy_min= 0 peak_at=02:17Z
|
||||
2026-09-25 max1= 2.13 best5= 0.67 busy_min= 0 peak_at=02:18Z
|
||||
2026-09-26 max1= 1.77 best5= 0.63 busy_min= 0 peak_at=02:18Z
|
||||
2026-09-27 max1= 2.04 best5= 0.75 busy_min= 0 peak_at=02:18Z
|
||||
2026-09-28 max1= 17.79 best5= 6.89 busy_min= 3 peak_at=13:43Z
|
||||
2026-09-29 max1= 2.45 best5= 0.65 busy_min= 0 peak_at=02:17Z
|
||||
2026-09-30 max1= 2.62 best5= 0.76 busy_min= 0 peak_at=02:18Z
|
||||
2026-10-01 max1= 1.02 best5= 0.54 busy_min= 0 peak_at=02:18Z
|
||||
2026-10-02 max1= 2.02 best5= 0.62 busy_min= 0 peak_at=02:18Z
|
||||
2026-10-03 max1= 2.01 best5= 0.60 busy_min= 0 peak_at=15:25Z
|
||||
2026-10-04 max1= 2.08 best5= 0.42 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-05 max1= 2.36 best5= 0.48 busy_min= 0 peak_at=05:26Z
|
||||
2026-10-06 max1= 1.58 best5= 0.32 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-07 max1= 1.05 best5= 0.30 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-08 max1= 1.43 best5= 0.29 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-09 max1= 1.47 best5= 0.29 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-10 max1= 0.90 best5= 0.27 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-11 max1= 1.34 best5= 0.30 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-12 max1= 4.96 best5= 0.99 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-13 max1= 1.31 best5= 0.28 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-14 max1= 6.31 best5= 1.27 busy_min= 1 peak_at=02:18Z
|
||||
2026-10-15 max1= 2.25 best5= 0.45 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-16 max1= 2.78 best5= 0.66 busy_min= 0 peak_at=20:55Z
|
||||
2026-10-17 max1= 2.26 best5= 0.45 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-18 max1= 2.08 best5= 0.47 busy_min= 0 peak_at=02:18Z
|
||||
2026-10-19 max1= 16.95 best5= 5.08 busy_min= 2 peak_at=02:18Z
|
||||
2026-10-20 max1= 1.20 best5= 0.48 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-21 max1= 2.27 best5= 0.46 busy_min= 0 peak_at=02:18Z
|
||||
2026-10-22 max1= 2.13 best5= 0.52 busy_min= 0 peak_at=02:18Z
|
||||
2026-10-23 max1= 1.37 best5= 0.48 busy_min= 0 peak_at=02:18Z
|
||||
nights with a push (best5>2): 2; best5 min/median/max = 5.1/6.9/6.9 Mbit/s
|
||||
|
||||
== demo-felhom: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
|
||||
2026-09-23 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=02:15Z
|
||||
2026-09-24 max1= 0.06 best5= 0.01 busy_min= 0 peak_at=02:15Z
|
||||
2026-09-25 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||
2026-09-26 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||
2026-09-27 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=00:19Z
|
||||
2026-09-28 max1= 0.03 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||
2026-09-29 max1= 0.03 best5= 0.01 busy_min= 0 peak_at=02:15Z
|
||||
2026-09-30 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-01 max1= 0.07 best5= 0.01 busy_min= 0 peak_at=19:54Z
|
||||
2026-10-02 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-03 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-04 max1= 0.02 best5= 0.00 busy_min= 0 peak_at=06:58Z
|
||||
2026-10-05 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=05:21Z
|
||||
2026-10-06 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=17:34Z
|
||||
2026-10-07 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-08 max1= 0.09 best5= 0.02 busy_min= 0 peak_at=13:04Z
|
||||
2026-10-09 max1= 0.10 best5= 0.02 busy_min= 0 peak_at=05:34Z
|
||||
2026-10-10 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-11 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=10:46Z
|
||||
2026-10-12 max1= 0.08 best5= 0.02 busy_min= 0 peak_at=06:18Z
|
||||
2026-10-13 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=13:54Z
|
||||
2026-10-14 max1= 0.07 best5= 0.02 busy_min= 0 peak_at=07:59Z
|
||||
2026-10-15 max1= 0.14 best5= 0.03 busy_min= 0 peak_at=14:25Z
|
||||
2026-10-16 max1= 0.84 best5= 0.17 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-17 max1= 0.49 best5= 0.10 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-18 max1= 0.33 best5= 0.07 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-19 max1= 1.73 best5= 0.35 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-20 max1= 0.23 best5= 0.07 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-21 max1= 0.36 best5= 0.07 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-22 max1= 0.52 best5= 0.11 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-23 max1= 0.48 best5= 0.10 busy_min= 0 peak_at=02:15Z
|
||||
|
||||
== tester-1: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
|
||||
2026-10-04 max1= 0.17 best5= 0.14 busy_min= 0 peak_at=19:55Z
|
||||
2026-10-05 max1= 1.44 best5= 0.29 busy_min= 0 peak_at=03:10Z
|
||||
2026-10-06 max1= 0.64 best5= 0.13 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-07 max1= 0.64 best5= 0.13 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-08 max1= 0.42 best5= 0.08 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-09 max1= 0.48 best5= 0.10 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-10 max1= 0.84 best5= 0.17 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-11 max1= 0.46 best5= 0.09 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-12 max1= 1.71 best5= 0.34 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-13 max1= 0.99 best5= 0.20 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-14 max1= 0.45 best5= 0.09 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-15 max1= 37.47 best5= 7.58 busy_min= 1 peak_at=02:17Z
|
||||
2026-10-16 max1= 3.85 best5= 0.77 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-17 max1= 5.11 best5= 1.21 busy_min= 1 peak_at=02:16Z
|
||||
2026-10-18 max1= 0.50 best5= 0.10 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-19 max1= 5.14 best5= 1.03 busy_min= 1 peak_at=02:16Z
|
||||
2026-10-20 max1= 0.52 best5= 0.10 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-21 max1= 0.61 best5= 0.12 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-22 max1= 0.53 best5= 0.11 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-23 max1= 0.80 best5= 0.18 busy_min= 0 peak_at=02:16Z
|
||||
nights with a push (best5>2): 1; best5 min/median/max = 7.6/7.6/7.6 Mbit/s
|
||||
@@ -0,0 +1,6 @@
|
||||
login 204 cookie
|
||||
{"t": 0.2, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65136369198, "engine_tx": 65136367684, "cli": "65.5MB / 65.1GB"}
|
||||
{"t": 6.5, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65152041285, "engine_tx": 65152039771, "cli": "65.9MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 19.9, "rate_engine_Mbit": 19.9}
|
||||
{"t": 12.7, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65167646902, "engine_tx": 65167645388, "cli": "66.2MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 20.14, "rate_engine_Mbit": 20.14}
|
||||
{"t": 19.0, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65183320503, "engine_tx": 65183318989, "cli": "66.6MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 19.9, "rate_engine_Mbit": 19.9}
|
||||
{"t": 25.2, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65198925974, "engine_tx": 65198924460, "cli": "66.9MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 20.14, "rate_engine_Mbit": 20.14}
|
||||
@@ -0,0 +1,10 @@
|
||||
LISTEN 0 5 0.0.0.0:5001 0.0.0.0:*
|
||||
login 204 cookie
|
||||
{"t": 0.0, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 1559315619, "engine_tx": 1558369279, "cli": "2.94MB / 2.85GB"}
|
||||
{"t": 6.0, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 9161893432, "engine_tx": 9161333668, "cli": "9.34MB / 10.5GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10136.77, "rate_engine_Mbit": 10137.29}
|
||||
{"t": 12.1, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 17020963755, "engine_tx": 17019975365, "cli": "16.7MB / 18.7GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10306.98, "rate_engine_Mbit": 10306.42}
|
||||
{"t": 18.1, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 25140960556, "engine_tx": 25139940620, "cli": "23.6MB / 26.5GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10826.66, "rate_engine_Mbit": 10826.62}
|
||||
{"t": 24.2, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 33049634829, "engine_tx": 33048966257, "cli": "30.6MB / 34.4GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10372.03, "rate_engine_Mbit": 10372.49}
|
||||
{"t": 30.2, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 41055773948, "engine_tx": 41055105508, "cli": "37.6MB / 42.3GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10674.85, "rate_engine_Mbit": 10674.85}
|
||||
{"t": 36.3, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 48735514801, "engine_tx": 48734576785, "cli": "44.3MB / 50GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10071.79, "rate_engine_Mbit": 10071.44}
|
||||
{"t": 42.3, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 56370706217, "engine_tx": 56370119481, "cli": "52.4MB / 58.8GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10180.26, "rate_engine_Mbit": 10180.72}
|
||||
@@ -0,0 +1,9 @@
|
||||
login 204 cookie
|
||||
setUploadLimit (200, '')
|
||||
uploadLimit now 2499584 prefs.up_limit 2499584
|
||||
login 204 cookie
|
||||
{"t": 0.0, "qb_up_speed": 16386315, "qb_up_data": 1346633728, "qb_tor_uploaded": 1346633728, "proc_tx": 1422747831, "engine_tx": 1422747831, "cli": "27.2MB / 1.42GB"}
|
||||
{"t": 6.0, "qb_up_speed": 0, "qb_up_data": 1374387364, "qb_tor_uploaded": 1374387364, "proc_tx": 1434635707, "engine_tx": 1434635707, "cli": "27.6MB / 1.44GB", "rate_qb_data_Mbit": 37.0, "rate_proc_Mbit": 15.85, "rate_engine_Mbit": 15.85}
|
||||
{"t": 12.1, "qb_up_speed": 2872275, "qb_up_data": 1392093322, "qb_tor_uploaded": 1391692420, "proc_tx": 1457229455, "engine_tx": 1457229455, "cli": "27.9MB / 1.46GB", "rate_qb_data_Mbit": 23.22, "rate_proc_Mbit": 29.63, "rate_engine_Mbit": 29.63}
|
||||
{"t": 18.1, "qb_up_speed": 1975559, "qb_up_data": 1406540742, "qb_tor_uploaded": 1406540742, "proc_tx": 1471460371, "engine_tx": 1471460371, "cli": "28.2MB / 1.47GB", "rate_qb_data_Mbit": 19.26, "rate_proc_Mbit": 18.97, "rate_engine_Mbit": 18.97}
|
||||
{"t": 24.2, "qb_up_speed": 3254241, "qb_up_data": 1423113257, "qb_tor_uploaded": 1420865709, "proc_tx": 1488294067, "engine_tx": 1488294067, "cli": "28.5MB / 1.49GB", "rate_qb_data_Mbit": 21.73, "rate_proc_Mbit": 22.08, "rate_engine_Mbit": 22.08}
|
||||
@@ -0,0 +1,11 @@
|
||||
login 204 cookie
|
||||
{"t": 0.0, "qb_up_speed": 16266049, "qb_up_data": 117833728, "qb_tor_uploaded": 117701484, "proc_tx": 123161204, "engine_tx": 123161204, "cli": "6.74MB / 140MB"}
|
||||
{"t": 6.0, "qb_up_speed": 16384087, "qb_up_data": 216137728, "qb_tor_uploaded": 216137728, "proc_tx": 225760308, "engine_tx": 225760308, "cli": "8.38MB / 243MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 136.8, "rate_engine_Mbit": 136.8}
|
||||
{"t": 12.1, "qb_up_speed": 16387266, "qb_up_data": 314441728, "qb_tor_uploaded": 314441728, "proc_tx": 328359626, "engine_tx": 328359626, "cli": "10.1MB / 346MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 134.56, "rate_engine_Mbit": 134.56}
|
||||
{"t": 18.1, "qb_up_speed": 16391332, "qb_up_data": 412745728, "qb_tor_uploaded": 412745728, "proc_tx": 431576783, "engine_tx": 431540471, "cli": "11.8MB / 450MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 137.62, "rate_engine_Mbit": 137.57}
|
||||
{"t": 24.2, "qb_up_speed": 16377743, "qb_up_data": 511049728, "qb_tor_uploaded": 511049728, "proc_tx": 536615588, "engine_tx": 536559607, "cli": "13.4MB / 556MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 137.76, "rate_engine_Mbit": 137.73}
|
||||
{"t": 30.2, "qb_up_speed": 16387474, "qb_up_data": 609353728, "qb_tor_uploaded": 609353728, "proc_tx": 641957804, "engine_tx": 641903336, "cli": "14.9MB / 660MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 140.46, "rate_engine_Mbit": 140.46}
|
||||
{"t": 36.3, "qb_up_speed": 16383934, "qb_up_data": 707657728, "qb_tor_uploaded": 707657728, "proc_tx": 746552576, "engine_tx": 746552576, "cli": "16.8MB / 764MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 137.17, "rate_engine_Mbit": 137.24}
|
||||
{"t": 42.3, "qb_up_speed": 16392852, "qb_up_data": 805961728, "qb_tor_uploaded": 805961728, "proc_tx": 849903723, "engine_tx": 849903723, "cli": "18.3MB / 867MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 137.8, "rate_engine_Mbit": 137.8}
|
||||
{"t": 48.4, "qb_up_speed": 16382132, "qb_up_data": 904265728, "qb_tor_uploaded": 904265728, "proc_tx": 952502020, "engine_tx": 952502020, "cli": "19.9MB / 970MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 134.56, "rate_engine_Mbit": 134.56}
|
||||
{"t": 54.4, "qb_up_speed": 16376106, "qb_up_data": 1002569728, "qb_tor_uploaded": 1002569728, "proc_tx": 1055099637, "engine_tx": 1055099637, "cli": "21.5MB / 1.07GB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 136.8, "rate_engine_Mbit": 136.8}
|
||||
@@ -0,0 +1,8 @@
|
||||
{'arguments': {}, 'result': 'success'}
|
||||
{'speed-limit-up': 2500, 'speed-limit-up-enabled': True}
|
||||
{"t": 0.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157242728}
|
||||
{"t": 5.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157244479, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 10.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157246230, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 15.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157248133, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157249884, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 25.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157251635, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
@@ -0,0 +1,10 @@
|
||||
{"t": 0.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157256302}
|
||||
{"t": 10.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157258205, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157261594, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 30.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157263345, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 40.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157265096, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 50.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157268485, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 60.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157271188, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 70.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157272939, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 80.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157276328, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 90.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157278588, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
@@ -0,0 +1,15 @@
|
||||
login 204 cookie
|
||||
/api/v2/torrents/stop (200, '')
|
||||
{'arguments': {}, 'result': 'success'}
|
||||
{"t": 0.0, "tr_upspeed": 40648163, "tr_cur_up": 1528429958, "tr_tor_up": 1528429958, "tx": 1534546958}
|
||||
{"t": 5.0, "tr_upspeed": 65588000, "tr_cur_up": 1832873057, "tr_tor_up": 1832873057, "tx": 1839315356, "rate_tr_Mbit": 487.11, "rate_container_Mbit": 487.63}
|
||||
{"t": 10.0, "tr_upspeed": 23587084, "tr_cur_up": 2076811226, "tr_tor_up": 2076811226, "tx": 2083512878, "rate_tr_Mbit": 390.3, "rate_container_Mbit": 390.72}
|
||||
{"t": 15.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157213873, "rate_tr_Mbit": 115.8, "rate_container_Mbit": 117.92}
|
||||
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157215624, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 25.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157218947, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.01}
|
||||
{"t": 30.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157220698, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 35.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157222449, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 40.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157224200, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 45.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157225951, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 50.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157227702, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 55.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157231243, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.01}
|
||||
@@ -0,0 +1,2 @@
|
||||
# usage: gen.sh <dir> ; writes <dir>/uwtest.bin, 2 GiB, identical on every machine (AES-CTR keystream of a fixed key)
|
||||
mkdir -p "$1"; openssl enc -aes-128-ctr -nosalt -K 00112233445566778899aabbccddeeff -iv 0 -in /dev/zero 2>/dev/null | head -c 2147483648 > "$1/uwtest.bin"; sha256sum "$1/uwtest.bin" | cut -c1-16
|
||||
@@ -0,0 +1,30 @@
|
||||
import sqlite3,sys,collections,datetime,statistics
|
||||
def pct(a,p):
|
||||
if not a: return 0
|
||||
a=sorted(a); k=(len(a)-1)*p/100; f=int(k); c=min(f+1,len(a)-1); return a[f]+(a[c]-a[f])*(k-f)
|
||||
for box in sys.argv[1:]:
|
||||
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
|
||||
lo,hi,n=db.execute('select min(ts),max(ts),count(*) from container_metrics').fetchone()
|
||||
print(f'\n=== {box.rsplit("/",1)[1]}: {datetime.datetime.utcfromtimestamp(lo):%Y-%m-%d %H:%M} -> {datetime.datetime.utcfromtimestamp(hi):%Y-%m-%d %H:%M} UTC ({(hi-lo)/86400:.1f} d), {n} rows')
|
||||
rows=collections.defaultdict(list)
|
||||
for ts,name,cpu,tx in db.execute('select ts,container_name,cpu_percent,net_tx_bytes from container_metrics order by ts'):
|
||||
rows[name].append((ts,cpu,tx))
|
||||
hour=collections.defaultdict(float)
|
||||
out=[]
|
||||
for name,r in rows.items():
|
||||
rates=[];resets=0;zero=all((x[2] or 0)==0 for x in r)
|
||||
for (t0,_,a),(t1,_,b) in zip(r,r[1:]):
|
||||
if a is None or b is None or t1-t0<=0 or t1-t0>180: continue
|
||||
d=b-a
|
||||
if d<0: resets+=1; continue
|
||||
mbps=d*8/(t1-t0)/1e6; rates.append(mbps)
|
||||
hour[datetime.datetime.utcfromtimestamp(t1).hour]+=d
|
||||
cpus=[x[1] for x in r]
|
||||
out.append((pct(rates,95),name,len(r),pct(rates,50),pct(rates,95),max(rates or [0]),pct(cpus,50),pct(cpus,95),max(cpus),zero,resets))
|
||||
print(f'{"container":32} {"n":>6} {"tx p50":>8} {"p95":>8} {"max":>8} Mbit/s | {"cpu p50":>7} {"p95":>7} {"max":>7} %core | notes')
|
||||
for o in sorted(out,reverse=True):
|
||||
_,name,nn,a,b,c,d,e,f,zero,res=o
|
||||
note=('tx counter always 0 (host network?)' if zero else '')+(f' resets={res}' if res else '')
|
||||
print(f'{name[:32]:32} {nn:6} {a:8.3f} {b:8.3f} {c:8.2f} | {d:7.1f} {e:7.1f} {f:7.1f} | {note}')
|
||||
tot=sum(hour.values()) or 1
|
||||
print('busiest UTC hours by bytes sent:',', '.join(f'{h:02d}h {100*v/tot:.0f}%' for h,v in sorted(hour.items(),key=lambda x:-x[1])[:4]))
|
||||
@@ -0,0 +1,21 @@
|
||||
import sqlite3,sys,collections,datetime
|
||||
for box in sys.argv[1:]:
|
||||
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
|
||||
r=db.execute("select ts,net_tx_bytes from container_metrics where container_name='felhom-controller' order by ts").fetchall()
|
||||
per=collections.defaultdict(list)
|
||||
for (t0,a),(t1,b) in zip(r,r[1:]):
|
||||
if a is None or b is None or b<a or not 0<t1-t0<=180: continue
|
||||
d=datetime.datetime.utcfromtimestamp(t1)
|
||||
per[d.date()].append((t1,(b-a)*8/(t1-t0)/1e6))
|
||||
print(f'\n== {box.rsplit("/",1)[1]}: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak')
|
||||
peaks=[]
|
||||
for day in sorted(per):
|
||||
v=per[day]; rates=[x[1] for x in v]
|
||||
best5=max((sum(rates[i:i+5])/5 for i in range(max(1,len(rates)-4))),default=0)
|
||||
m=max(v,key=lambda x:x[1])
|
||||
busy=sum(1 for x in rates if x>5)
|
||||
peaks.append(best5)
|
||||
print(f'{day} max1={m[1]:6.2f} best5={best5:6.2f} busy_min={busy:4d} peak_at={datetime.datetime.utcfromtimestamp(m[0]):%H:%M}Z')
|
||||
big=[p for p in peaks if p>2]
|
||||
if big:
|
||||
big.sort(); print(f'nights with a push (best5>2): {len(big)}; best5 min/median/max = {big[0]:.1f}/{big[len(big)//2]:.1f}/{big[-1]:.1f} Mbit/s')
|
||||
@@ -0,0 +1,10 @@
|
||||
#!/bin/bash
|
||||
# Waits until 02:08Z on 2026-10-24, then reads the controller container's eth0 tx bytes once a second for 22 minutes on
|
||||
# each of the three test boxes (read only: /proc/<pid>/net/dev inside guest 9201). Output: one file per box.
|
||||
OUT=$1
|
||||
target=$(date -u -d '2026-10-24T02:08:00Z' +%s); now=$(date -u +%s); [ $target -gt $now ] && sleep $((target-now))
|
||||
INNER='P=$(docker inspect -f {{.State.Pid}} felhom-controller); end=$(( $(date +%s) + 1320 )); while [ $(date +%s) -lt $end ]; do echo "$(date +%s.%N | cut -c1-14) $(awk "/eth0:/{print \$10}" /proc/$P/net/dev)"; sleep 1; done'
|
||||
for pair in "demo-hp:demo-hp" "felhom-pve:demo-felhom" "root@192.168.0.154:tester-1"; do h=${pair%%:*}; n=${pair##*:}
|
||||
ssh -o BatchMode=yes -o ServerAliveInterval=30 $h "pct exec 9201 -- bash -c '$INNER'" > "$OUT/night-push-1s-$n.txt" 2>&1 &
|
||||
done
|
||||
wait; echo "done $(date -u +%FT%TZ)"
|
||||
@@ -0,0 +1,89 @@
|
||||
# Runs INSIDE guest 9202. Asks qBittorrent's own Web API and reads the container's exact byte counters.
|
||||
# The admin password is read from the container env at use (as the controller does) and never printed.
|
||||
import json, subprocess, sys, time, urllib.request, urllib.parse, http.client, socket, re, uuid
|
||||
def sh(*a): return subprocess.run(a, capture_output=True, text=True).stdout.strip()
|
||||
C = sys.argv[1] if len(sys.argv) > 1 else 'qbittorrent'
|
||||
IP = sh('docker', 'inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}', C).split()[0]
|
||||
PID = sh('docker', 'inspect', '-f', '{{.State.Pid}}', C)
|
||||
CID = sh('docker', 'inspect', '-f', '{{.Id}}', C)
|
||||
BASE = f'http://{IP}:8080'
|
||||
PW = sh('docker', 'exec', C, 'printenv', 'QBT_ADMIN_PASSWORD')
|
||||
def req(method, path, data=None, hdr=None, raw=False):
|
||||
h = {'Referer': BASE}
|
||||
h.update(hdr or {})
|
||||
if COOKIE: h['Cookie'] = COOKIE
|
||||
r = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(r, timeout=20) as resp:
|
||||
b = resp.read(); return resp.status, (b if raw else (json.loads(b) if b and b[:1] in b'[{' else b.decode()))
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code, e.read().decode()[:200]
|
||||
COOKIE = None
|
||||
code, _ = None, None
|
||||
r = urllib.request.Request(BASE + '/api/v2/auth/login', data=urllib.parse.urlencode({'username': 'admin', 'password': PW}).encode(),
|
||||
headers={'Referer': BASE, 'Content-Type': 'application/x-www-form-urlencoded'}, method='POST')
|
||||
with urllib.request.urlopen(r, timeout=20) as resp:
|
||||
m = re.search(r'((?:QBT_)?SID[A-Za-z0-9_]*)=([^;]+)', '; '.join(resp.headers.get_all('Set-Cookie') or []))
|
||||
print('login', resp.status, 'cookie' if m else 'NO COOKIE')
|
||||
COOKIE = f'{m.group(1)}={m.group(2)}'
|
||||
def netdev():
|
||||
for line in open(f'/proc/{PID}/net/dev'):
|
||||
if line.strip().startswith('eth0:'):
|
||||
f = line.split(':', 1)[1].split(); return int(f[8])
|
||||
def engine_stats():
|
||||
s = socket.socket(socket.AF_UNIX); s.connect('/var/run/docker.sock')
|
||||
s.sendall(f'GET /containers/{CID}/stats?stream=false&one-shot=true HTTP/1.0\r\n\r\n'.encode())
|
||||
b = b''
|
||||
while True:
|
||||
x = s.recv(65536)
|
||||
if not x: break
|
||||
b += x
|
||||
j = json.loads(b.split(b'\r\n\r\n', 1)[1])
|
||||
return sum(n['tx_bytes'] for n in j.get('networks', {}).values()), j['cpu_stats']['cpu_usage']['total_usage']
|
||||
def cli_stats():
|
||||
return sh('docker', 'stats', '--no-stream', '--format', '{{.NetIO}}', C)
|
||||
mode = sys.argv[2] if len(sys.argv) > 2 else 'info'
|
||||
if mode == 'info':
|
||||
print('version', req('GET', '/api/v2/app/version')[1], 'webapi', req('GET', '/api/v2/app/webapiVersion')[1])
|
||||
print('transfer/info', req('GET', '/api/v2/transfer/info')[1])
|
||||
p = req('GET', '/api/v2/app/preferences')[1]
|
||||
print('prefs', {k: p.get(k) for k in ('up_limit', 'dl_limit', 'alt_up_limit', 'alt_dl_limit', 'scheduler_enabled', 'limit_tcp_overhead', 'limit_lan_peers', 'limit_utp_rate', 'max_uploads', 'enable_upload_suggestions')})
|
||||
print('uploadLimit', req('GET', '/api/v2/transfer/uploadLimit')[1], 'speedLimitsMode', req('GET', '/api/v2/transfer/speedLimitsMode')[1])
|
||||
elif mode == 'add':
|
||||
bnd = uuid.uuid4().hex
|
||||
tor = open('/root/uwtest.torrent', 'rb').read()
|
||||
parts = []
|
||||
for k, v in (('savepath', '/mnt/felhom-drives/scratch_hdd/userdata/media/torrents'), ('skip_checking', 'true'), ('category', '')):
|
||||
parts.append(f'--{bnd}\r\nContent-Disposition: form-data; name="{k}"\r\n\r\n{v}\r\n'.encode())
|
||||
parts.append(f'--{bnd}\r\nContent-Disposition: form-data; name="torrents"; filename="uwtest.torrent"\r\nContent-Type: application/x-bittorrent\r\n\r\n'.encode() + tor + b'\r\n')
|
||||
parts.append(f'--{bnd}--\r\n'.encode())
|
||||
print('add', req('POST', '/api/v2/torrents/add', b''.join(parts), {'Content-Type': f'multipart/form-data; boundary={bnd}'}))
|
||||
time.sleep(3)
|
||||
print('torrents', [(t['name'], t['state'], t['progress']) for t in req('GET', '/api/v2/torrents/info')[1]])
|
||||
elif mode == 'setlimit':
|
||||
print('setUploadLimit', req('POST', '/api/v2/transfer/setUploadLimit', urllib.parse.urlencode({'limit': sys.argv[3]}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||
print('uploadLimit now', req('GET', '/api/v2/transfer/uploadLimit')[1], 'prefs.up_limit', req('GET', '/api/v2/app/preferences')[1].get('up_limit'))
|
||||
elif mode == 'setpref':
|
||||
print('setPreferences', req('POST', '/api/v2/app/setPreferences', urllib.parse.urlencode({'json': json.dumps({'up_limit': int(sys.argv[3])})}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||
print('uploadLimit now', req('GET', '/api/v2/transfer/uploadLimit')[1], 'prefs.up_limit', req('GET', '/api/v2/app/preferences')[1].get('up_limit'))
|
||||
elif mode == 'peer':
|
||||
print('addPeers', req('POST', '/api/v2/torrents/addPeers', urllib.parse.urlencode({'hashes': 'all', 'peers': sys.argv[3]}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||
print('reannounce', req('POST', '/api/v2/torrents/reannounce', urllib.parse.urlencode({'hashes': 'all'}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||
elif mode == 'post':
|
||||
print(sys.argv[3], req('POST', sys.argv[3], urllib.parse.urlencode(json.loads(sys.argv[4])).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||
elif mode == 'sample':
|
||||
dur, step = int(sys.argv[3]), int(sys.argv[4])
|
||||
t0 = time.time(); prev = None
|
||||
while time.time() - t0 <= dur:
|
||||
ti = req('GET', '/api/v2/transfer/info')[1]
|
||||
tor = req('GET', '/api/v2/torrents/info')[1]
|
||||
etx, ecpu = engine_stats()
|
||||
row = {'t': round(time.time() - t0, 1), 'qb_up_speed': ti['up_info_speed'], 'qb_up_data': ti['up_info_data'],
|
||||
'qb_tor_uploaded': sum(x.get('uploaded', 0) for x in tor), 'proc_tx': netdev(), 'engine_tx': etx, 'cli': cli_stats()}
|
||||
if prev:
|
||||
dt = row['t'] - prev['t']
|
||||
row['rate_qb_data_Mbit'] = round((row['qb_up_data'] - prev['qb_up_data']) * 8 / dt / 1e6, 2)
|
||||
row['rate_proc_Mbit'] = round((row['proc_tx'] - prev['proc_tx']) * 8 / dt / 1e6, 2)
|
||||
row['rate_engine_Mbit'] = round((row['engine_tx'] - prev['engine_tx']) * 8 / dt / 1e6, 2)
|
||||
print(json.dumps(row), flush=True); prev = row
|
||||
time.sleep(step)
|
||||
@@ -0,0 +1,32 @@
|
||||
import sqlite3,sys,collections
|
||||
def runs(r, pred):
|
||||
best=cur=0; bt=None; prev=None
|
||||
for t,v in r:
|
||||
if pred(v) and prev is not None and t-prev<=180: cur+=1
|
||||
elif pred(v): cur=1
|
||||
else: cur=0
|
||||
if cur>best: best=cur; bt=t
|
||||
prev=t
|
||||
return best,bt
|
||||
import datetime
|
||||
for box in sys.argv[1:]:
|
||||
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
|
||||
nc=None
|
||||
rows=collections.defaultdict(list)
|
||||
for ts,name,cpu,tx in db.execute('select ts,container_name,cpu_percent,net_tx_bytes from container_metrics order by ts'):
|
||||
rows[name].append((ts,cpu,tx))
|
||||
print('==',box.rsplit('/',1)[1])
|
||||
out=[]
|
||||
for name,r in rows.items():
|
||||
cpu=[(t,c) for t,c,_ in r]
|
||||
tx=[]
|
||||
for (t0,_,a),(t1,_,b) in zip(r,r[1:]):
|
||||
if a is None or b is None or b<a or not 0<t1-t0<=180: continue
|
||||
tx.append((t1,(b-a)*8/(t1-t0)/1e6))
|
||||
c150,ct=runs(cpu,lambda v:v>=150); c80,_=runs(cpu,lambda v:v>=80)
|
||||
u5,ut=runs(tx,lambda v:v>=5); u1,_=runs(tx,lambda v:v>=1)
|
||||
if max(c80,u1)>=3: out.append((name,c80,c150,ct,u1,u5,ut))
|
||||
print(f'{"container":28} cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]')
|
||||
for name,c80,c150,ct,u1,u5,ut in sorted(out,key=lambda x:-max(x[1],x[4])):
|
||||
f=lambda t: datetime.datetime.utcfromtimestamp(t).strftime('%m-%d %H:%M') if t else '-'
|
||||
print(f'{name[:28]:28} {c80:8} {c150:9} ({f(ct)}) {u1:9} {u5:9} ({f(ut)})')
|
||||
@@ -0,0 +1,21 @@
|
||||
# Minimal BitTorrent HTTP tracker for a closed scratch test (no internet). Compact peer lists.
|
||||
import http.server, urllib.parse, socket, struct, time
|
||||
peers = {} # info_hash -> {(ip,port): last_seen}
|
||||
def benc(o):
|
||||
if isinstance(o, int): return b"i%de" % o
|
||||
if isinstance(o, bytes): return b"%d:%s" % (len(o), o)
|
||||
if isinstance(o, str): return benc(o.encode())
|
||||
if isinstance(o, dict): return b"d" + b"".join(benc(k) + benc(v) for k, v in sorted(o.items())) + b"e"
|
||||
class H(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
q = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query.encode('latin-1').decode('latin-1'), encoding='latin-1')
|
||||
ih = q.get('info_hash', [''])[0]; port = int(q.get('port', ['0'])[0]); ip = self.client_address[0]
|
||||
d = peers.setdefault(ih, {}); now = time.time()
|
||||
if q.get('event', [''])[0] == 'stopped': d.pop((ip, port), None)
|
||||
else: d[(ip, port)] = now
|
||||
lst = b"".join(socket.inet_aton(i) + struct.pack(">H", p) for (i, p), t in d.items() if now - t < 600 and (i, p) != (ip, port))
|
||||
body = benc({"interval": 30, "peers": lst})
|
||||
print(time.strftime('%H:%M:%S'), 'announce', ip, port, q.get('event', [''])[0], 'peers', len(lst) // 6, flush=True)
|
||||
self.send_response(200); self.send_header('Content-Length', str(len(body))); self.end_headers(); self.wfile.write(body)
|
||||
def log_message(self, *a): pass
|
||||
http.server.ThreadingHTTPServer(('0.0.0.0', 6969), H).serve_forever()
|
||||
@@ -0,0 +1,47 @@
|
||||
# Runs INSIDE guest 9202. Transmission's own RPC numbers vs the container's exact tx bytes. Password read at use, never printed.
|
||||
import json, subprocess, sys, time, urllib.request, base64
|
||||
def sh(*a): return subprocess.run(a, capture_output=True, text=True).stdout.strip()
|
||||
C = 'transmission'
|
||||
IP = sh('docker', 'inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}', C).split()[0]
|
||||
PID = sh('docker', 'inspect', '-f', '{{.State.Pid}}', C)
|
||||
PW = sh('docker', 'exec', C, 'printenv', 'PASS')
|
||||
AUTH = 'Basic ' + base64.b64encode(('admin:' + PW).encode()).decode()
|
||||
SID = ''
|
||||
def rpc(method, args=None):
|
||||
global SID
|
||||
for _ in range(2):
|
||||
r = urllib.request.Request(f'http://{IP}:9091/transmission/rpc', data=json.dumps({'method': method, 'arguments': args or {}}).encode(),
|
||||
headers={'Authorization': AUTH, 'Host': 'transmission:9091', 'X-Transmission-Session-Id': SID, 'Content-Type': 'application/json'}, method='POST')
|
||||
try:
|
||||
with urllib.request.urlopen(r, timeout=20) as resp: return json.loads(resp.read())
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 409: SID = e.headers.get('X-Transmission-Session-Id'); continue
|
||||
return {'http': e.code}
|
||||
def tx():
|
||||
for line in open(f'/proc/{PID}/net/dev'):
|
||||
if line.strip().startswith('eth0:'): return int(line.split(':', 1)[1].split()[8])
|
||||
mode = sys.argv[1]
|
||||
if mode == 'info':
|
||||
g = rpc('session-get')['arguments']
|
||||
print('version', g.get('version'), {k: g.get(k) for k in ('speed-limit-up', 'speed-limit-up-enabled', 'alt-speed-up', 'alt-speed-enabled', 'alt-speed-time-enabled', 'units')})
|
||||
print('session-stats', rpc('session-stats')['arguments'])
|
||||
elif mode == 'add':
|
||||
meta = base64.b64encode(open('/root/uwtest.torrent', 'rb').read()).decode()
|
||||
print(rpc('torrent-add', {'metainfo': meta, 'download-dir': '/mnt/felhom-drives/scratch_hdd/userdata/media/transmission', 'paused': False}))
|
||||
time.sleep(5); rpc('torrent-verify', {}); time.sleep(20)
|
||||
print(rpc('torrent-get', {'fields': ['name', 'status', 'percentDone', 'error', 'errorString']}))
|
||||
elif mode == 'limit':
|
||||
print(rpc('session-set', {'speed-limit-up': int(sys.argv[2]), 'speed-limit-up-enabled': True}))
|
||||
g = rpc('session-get')['arguments']; print({k: g.get(k) for k in ('speed-limit-up', 'speed-limit-up-enabled')})
|
||||
elif mode == 'reannounce':
|
||||
print(rpc('torrent-reannounce', {}))
|
||||
elif mode == 'sample':
|
||||
dur, step = int(sys.argv[2]), int(sys.argv[3]); t0 = time.time(); prev = None
|
||||
while time.time() - t0 <= dur:
|
||||
st = rpc('session-stats')['arguments']; tg = rpc('torrent-get', {'fields': ['uploadedEver', 'rateUpload']})['arguments']['torrents']
|
||||
row = {'t': round(time.time() - t0, 1), 'tr_upspeed': st['uploadSpeed'], 'tr_cur_up': st['current-stats']['uploadedBytes'], 'tr_tor_up': sum(x['uploadedEver'] for x in tg), 'tx': tx()}
|
||||
if prev:
|
||||
dt = row['t'] - prev['t']
|
||||
row['rate_tr_Mbit'] = round((row['tr_cur_up'] - prev['tr_cur_up']) * 8 / dt / 1e6, 2)
|
||||
row['rate_container_Mbit'] = round((row['tx'] - prev['tx']) * 8 / dt / 1e6, 2)
|
||||
print(json.dumps(row), flush=True); prev = row; time.sleep(step)
|
||||
@@ -0,0 +1,236 @@
|
||||
# Catalog sweep 2026-10-24 — every catalog app on a fresh box, the way a family uses it
|
||||
|
||||
Unattended validation sweep (Saturday 12:30Z → Sunday ~00:00Z). Brief: the operator's „every catalog app, one by one, on
|
||||
a fresh box" prompt. Venue: demo-hp (Tier 0). Method: endpoint-level — every step is the request the dashboard or the
|
||||
app's own page sends (no browser click-through except the screenshot harness; no `docker exec` to make state).
|
||||
|
||||
## Safest readings taken (the brief allows no question)
|
||||
|
||||
1. **ISO 1.30.0, not 1.36.0.** The brief said „1.36.0 or the current one"; the current one on felhom.eu/letoltes was
|
||||
1.30.0 (sha256 checked against the published `.sha256`). See `_box/README.md`.
|
||||
2. **12 GiB for the VM, not 16.** demo-hp had 14 GiB free beside 9201, 9202 and Tester 1; 12 is the brief's minimum.
|
||||
The guest itself is 7.7 GiB (the golden's size); the box's own memory admission decided what could run.
|
||||
3. **„Remove keeping data → install again with use my kept data" is a drive-folder question.** By design (`07` §6.5)
|
||||
the question is asked only for an app that keeps a folder on a drive (`HDD_PATH`). For the 40+ apps that keep their
|
||||
data in Docker volumes, „keep my data" keeps the recovery unit; the reinstall starts empty, and the item comes back by
|
||||
the unit restore („Visszaállítás"). So step 5 + 6 ran as: keep-remove → reinstall → read (empty, expected) → unit
|
||||
restore → read. That is a unit restore on **every** app with a seed, not only every fifth.
|
||||
4. **„Mentés most" is the local copy only** (`07` §8, by design); the second-drive and off-site copies run in the night.
|
||||
The sweep did not wait for a night, so „the right tiers ran" is checked as „the local unit holds the app".
|
||||
5. **Categories:** the brief names „Apps & catalog" for bigger issues; a row whose harm is in backup/restore or storage is
|
||||
filed in that category instead (`register_shape_gate` has one category per row, and the owner reads it there).
|
||||
6. **Sev:** the brief's scale (data at risk P1/P2, a broken app P2, a confusing step P3). R-1269, R-1271 are P2 by that rule (the data is not deleted in either; the household loses access to it).
|
||||
7. **The sweep box reads the LIVE catalog** (main `5e196e1`, then `227a544` — rules only, no template moved). The drill
|
||||
catalog was used only for the three apps whose fix needed proving; the box was pointed back after (`_drill-repoint.txt`).
|
||||
8. **Harness fixes live in the sweep's copy** (`tools/`), not in the catalog's scripts; porting them is R-1277.
|
||||
|
||||
## The box (Part 0)
|
||||
|
||||
`_box/README.md`: VM **9360** `sweep1024-box` (6 cores, 12 GiB, disks 256 + 200 + 150 GB on `nvme-scratch`), ISO 1.30.0,
|
||||
customer Tester4 (hub host `Tester4-aa9466`), guest 9201, domain `tester4.test` (no DNS: the browser LXC maps the
|
||||
names). Bind 12:42:18Z, day-0 SUCCESS 12:45:40Z, claim, recovery code escrowed (stored out-of-band), drives `hdd_1`
|
||||
(210 GB) and `hdd_2` (157 GB) plus the installer's internal place `belso`. Versions at the start: controller 0.342.0,
|
||||
agent 0.164.0, hub 0.169.0. Browser: LXC **9361** `sweep1024-browser` (Chromium + Playwright), 60 dashboard screens
|
||||
(15 pages × phone/computer × HU/EN) in `_box/screens-dashboard/`.
|
||||
|
||||
## Method per app (Part 1)
|
||||
|
||||
`tools/sweep.py <app>` — one app at a time, small first, heavy last (`tools/order.txt`); the next starts after the last
|
||||
is removed with its data. Each `<app>/` holds `record.json` (the measured facts), `walk.log`, the page texts
|
||||
(`page-{install,guide,overview}-{hu,en}.txt`) and `screens/` (phone + computer, HU + EN, install page and the app after
|
||||
first login). Steps: install through `POST /api/stacks/<app>/deploy` (timed to the front door answering); a stranger
|
||||
request without the family cookie (must be refused where the template gates); one item through the app's own API (the
|
||||
catalog's fixture where one exists, a new one in `tools/sweep_fixtures.py` for seven more); „Mentés most"; remove
|
||||
keeping data; reinstall (the kept-data question answered „use" where it is asked); read; unit restore; read; remove with
|
||||
data; leftovers checked (containers, volumes, `app.yaml`, `appdata`, `kept`, the subdomain → 404). Memory: the peak of
|
||||
each container's **anon** memory from its cgroup (`memory.stat`; the file cache is not counted — project rule) against
|
||||
its compose limit; a `*` in the table means only `docker stats` was read for that app (it includes the cache). The anon reading joined mid-sweep, so 22 apps carry it and 40 carry the `docker stats` upper bound; of those 40 none passed 79 % of its limit (Tandoor 1009/1280) and none restarted. Above 80 % on anon: only OpenGist (F14).
|
||||
|
||||
## Results
|
||||
|
||||
<!-- TABLE START -->
|
||||
| App | Verdict | Peak memory / limit (MiB) | Install → answering (s) | What we saw | Row / fix |
|
||||
|---|---|---|---|---|---|
|
||||
| actualbudget | **works** | 49* / 256 | 30 | the seeded item back after the restore | |
|
||||
| adventurelog | **works with a problem** | 381 / 896 | 207 | works; but the keep-backups → reinstall → restore path breaks it (DB password from the new install over the old data) — reproduced in its re-run | R-1269, R-1270 |
|
||||
| audiobookshelf | **works** | 56* / 512 | 41 | the seeded item back after the restore | drive names: R-1275 |
|
||||
| bentopdf | **works** | 3* / 384 | 30 | stateless (nothing to keep); installs, answers, removes clean | |
|
||||
| bookstack | **works** | 167* / 768 | 71 | the seeded item back after the restore | |
|
||||
| calcom | **works** | 807 / 1792 | 278 | the seeded item back after the restore; 2 start-up messages of the image itself | |
|
||||
| calibre-web | **works** | 211 / 768 | 76 | the book (its own upload) is listed again after the keep-data reinstall; the fixture's download check gets 401 (its sign-in for /download, not the app); the admin name is the box's generated one (re-run 2) | R-1277 |
|
||||
| claper | **works** | 267* / 640 | 86 | the seeded item back after the restore | |
|
||||
| code-server | **works** | 81 / 1024 | 46 | first install refused by the registry's rate limit, told as „exit code 1”; the re-run installed and worked (no seed: the editor has no API) | R-1273 |
|
||||
| crafty-controller | **works** | 79* / 2048 | 76 | the seeded item back after the restore | |
|
||||
| dawarich | **works** | 835* / 2688 | 136 | the seeded item back after the restore (every-5th: restore also before the reinstall) | |
|
||||
| docmost | **works** | 528* / 1152 | 126 | the seeded item back after the restore | |
|
||||
| emby | **works** | 95 / 2048 | 41 | installs, answers, removes clean; no seed (no fixture for its wizard) | |
|
||||
| ghost | **works** | 167* / 512 | 121 | the seeded item back after the restore | |
|
||||
| gitea | **works** | 112 / 512 | 25 | the seeded item back after the restore | |
|
||||
| glance | **works** | 7* / 128 | 15 | family gate holds a stranger; config-only (nothing to keep); the app's own headings are English | |
|
||||
| gokapi | **works** | 6 / 128 | 15 | installs, own login, removes clean; no seed (API key only in its UI) | |
|
||||
| grafana | **works** | 383* / 768 | 61 | the seeded item back after the restore | |
|
||||
| gramps-web | **works** | 347* / 1024 | 206 | installs, answers, removes clean; no seed fixture | |
|
||||
| grimmory | **works with a problem** | 594 / 1408 | 96 | first admin, library and an EPUB upload work; setup gate then family gate work as designed; the read-back through the family gate stayed inconclusive (the harness's gate logic, R-1277); remove-with-data leaves the empty parent folder | R-1276, R-1277 |
|
||||
| grocy | **works** | 34* / 384 | 36 | the seeded item back after the restore | |
|
||||
| home-assistant | **works** | 311 / 1024 | 161 | the seeded item back after the restore | |
|
||||
| homebox | **works** | 205 / 256 | 20 | the seeded item back after the restore | |
|
||||
| homepage | **works with a problem** | 91* / 256 | 25 | no gate at all: a stranger sees the box's CPU/RAM/disk (and later the family's links); still the upstream sample page | R-1272 |
|
||||
| immich | **works with a problem** | 1845 / 6016 | 121 | album (new fixture) in; „use my kept data” loaded the rehearsal's future-dated database and Immich never answered again (404, 20 min) | R-1271 |
|
||||
| jellyfin | **works** | 110 / 2048 | 41 | first user (new fixture, its wizard API) back after the restore | |
|
||||
| karakeep | **works** | 649* / 2816 | 107 | the seeded item back after the restore | |
|
||||
| kimai | **works** | 303* / 640 | 111 | own login; a Hungarian browser lands on /hu/ (curl without a language gets /en/) | |
|
||||
| komga | **works** | 473* / 768 | 46 | the seeded item back after the restore | drive names: R-1275 |
|
||||
| lubelogger | **works** | 60* / 512 | 25 | the seeded item back after the restore | |
|
||||
| mealie | **works** | 322 / 1000 | 72 | the seeded item back after the restore (the fixture raced after_install on the first run) | |
|
||||
| metube | **works** | 90* / 768 | 61 | family gate holds a stranger; download queue entry back | |
|
||||
| n8n | **works with a problem** | 346 / 1536 | 151 | the owner back after the restore; on the live template it tries to send usage data out ([Rudder] lines, 6 in the live re-run) — the drill fix: the variable in the running container, 0 error lines | drill a601950 (morning list) |
|
||||
| navidrome | **works** | 23* / 256 | 25 | the seeded item back after the restore | drive names: R-1275 |
|
||||
| nextcloud | **works with a problem** | 298 / 1664 | 122 | user in; „use my kept data” loaded the rehearsal's future-dated database beside today's files — the user was not there | R-1271 |
|
||||
| nzbget | **works** | 6* / 512 | 25 | the seeded item back after the restore; „no news server” until the family adds one (the guide says so) | |
|
||||
| onlyoffice | **works** | 464 / 2048 | 71 | document server (no household data to keep); installs, answers, removes clean (every-5th restore) | |
|
||||
| opengist | **works with a problem** | 125 / 128 | 15 | the gist owner back after the restore; but 128M is too small: one sign-up + sign-in hit the limit 443 times and swapped 98 MiB (memprobe-128M.txt); at 256M (drill) anon 216 MiB | drill 405077f (morning list) |
|
||||
| outline | **works** | 327 / 1152 | 151 | the seeded item back after the restore | |
|
||||
| paperless-ngx | **works with a problem** | 669* / 1664 | 146 | account in; „use my kept data” loaded the rehearsal's future-dated database — the account was not there | R-1271, R-1276 |
|
||||
| papra | **works** | 320 / 768 | 76 | the seeded item back after the restore (Papra 26.6 needs an Origin header — a browser sends it) | |
|
||||
| plant-it | **not installed** | – | – | lifecycle: abandoned — not offered for a new install (by design) | |
|
||||
| plex | **works** | 223 / 2048 | 30 | installs, answers, removes clean; no seed (a server is claimed with a plex.tv account) | |
|
||||
| privatebin | **works** | 31* / 128 | 15 | the seeded item back after the restore; D22 put-back PASS | |
|
||||
| prowlarr | **works** | 114* / 512 | 21 | the seeded item back after the restore | |
|
||||
| qbittorrent | **works** | 77* / 1024 | 15 | the seeded item back after the restore | |
|
||||
| radarr | **works** | 100* / 512 | 41 | the seeded item back after the restore | |
|
||||
| radicale | **works** | 33* / 128 | 25 | the seeded item back after the restore | |
|
||||
| rallly | **works** | 346 / 1024 | 71 | installs, answers, removes clean; no seed: Rallly 4.15 closed password sign-up (login by e-mail code); the guide's polls need no account | R-1277 |
|
||||
| recipe-importer | **works** | 67* / 128 | 25 | stateless helper; installs, answers, removes clean (every-5th restore) | |
|
||||
| romm | **works with a problem** | 748* / 1536 | 116 | item back through „use my kept data” (its own copy); remove-with-data left the empty parent folder | R-1276 |
|
||||
| sabnzbd | **works** | 69* / 1024 | 25 | the seeded item back after the restore | |
|
||||
| seerr | **works** | 166 / 384 | 92 | installs, answers (307 to its setup), removes clean; no seed (needs a media server) | |
|
||||
| sonarr | **works** | 80* / 512 | 41 | the seeded item back after the restore | |
|
||||
| sparkyfitness | **works** | 417* / 1792 | 91 | the seeded item back after the restore | |
|
||||
| tandoor | **works with a problem** | 1052* / 1536 | 177 | works; but the keep-backups → reinstall → restore path breaks it (DB password from the new install over the old data), and the restore still said success — reproduced on its own | R-1269, R-1270 |
|
||||
| termix | **works** | 111* / 256 | 61 | installs, answers, removes clean (every-5th restore); no seed fixture | |
|
||||
| transmission | **works** | 6* / 512 | 20 | the seeded item back after the restore | |
|
||||
| uptime-kuma | **works** | 120* / 384 | 76 | the seeded item back after the restore | |
|
||||
| vaultwarden | **works** | 37 / 256 | 25 | installs in 25 s, its own login page, sign-up closed (404); the box admin's invite works (run-live-2), but 1.37.4 has no route for the fixture's invited sign-up, so no item was seeded | R-1277 |
|
||||
| vikunja | **works** | 37* / 256 | 10 | the seeded item back after the restore; D22 put-back PASS | |
|
||||
| wanderer | **not testable on this box** | 140* / 1024 | 605 | not testable on this box: its web part calls its database at the PUBLIC name, which tester4.test cannot resolve (known, R-739) — unhealthy, 404 | R-739 |
|
||||
| wger | **not installed** | – | – | lifecycle: hidden — not offered for a new install (by design) | |
|
||||
| wishlist | **works** | 200* / 512 | 71 | the seeded item back after the restore | |
|
||||
| zipline | **works** | 233* / 768 | 81 | the seeded item back after the restore | |
|
||||
|
||||
works: 52, works with a problem: 10, not installed: 2, not testable on this box: 1 (of 65)
|
||||
<!-- TABLE END -->
|
||||
|
||||
## Findings — each a row, a fix or NOT-A-FINDING
|
||||
|
||||
| # | What | Disposition |
|
||||
|---|---|---|
|
||||
| F10 | A reinstall's own capture rewrites the unit that „remove — keep backups" kept; restoring it pairs the NEW install's DB password with the OLD data — the app never starts, the item is gone (Tandoor ×2, AdventureLog) | R-1269 (P2) |
|
||||
| F11 | The unit restore reports `ok: true` „…az alkalmazás újraindult" while the controller logged „health check failed … did not reach running state" | R-1270 (P2) |
|
||||
| F12 | „Use my kept data" loaded the rehearsal box's future-dated off-site database (2026-10-28) for Paperless-ngx, Nextcloud and Immich — another drive's install, which `07` §6.5 says is refused; Immich then never answered | R-1271 (P2) |
|
||||
| F1 | Homepage has no gate: a stranger sees the box's CPU/RAM/disk (Glance, same class, is gated) | R-1272 (operator decision) |
|
||||
| F7 | An install refused by the registry's rate limit (`toomanyrequests`) tells the household only „exit code 1" | R-1273 |
|
||||
| F8 | The restore refusal says „turn on remote backup, or connect a second drive" when both exist but have not copied yet | R-1274 |
|
||||
| F2 | English pages show the drive names in Hungarian („Belső tárhely", „Külső tárhely (hdd_1)") | R-1275 |
|
||||
| F9 | „Remove with data" leaves the empty parent folder `appdata/<app>` (Grimmory, RomM, Paperless-ngx) | R-1276 |
|
||||
| — | The catalog's box fixtures drifted from the apps (OpenGist, Calibre-Web, Mealie, Papra, Rallly; seven apps had none) | R-1277 |
|
||||
| F13 | n8n sends usage data (`[Rudder]` lines; no `N8N_DIAGNOSTICS_ENABLED=false`) | **fixed in the drill** `a601950`, proven below; morning list |
|
||||
| F14 | OpenGist at 125.8 of 128 MiB anon after one sign-up + sign-in; at 128M it hits the limit 443 times and swaps 98 MiB | **fixed in the drill** `405077f` + `492a17f` (256M), proven below; morning list |
|
||||
| — | 52 anonymous Docker volumes stayed after every app was removed with data (PrivateBin, OnlyOffice, RomM, Immich: the images' own `VOLUME` paths; 4–72 KB, no household data). The sweep's own leftover check filters by the app's label and missed them; found in the end-state read (`_box/anon-volumes.txt`) | known: note on R-1267 (widened to OnlyOffice and RomM) |
|
||||
| — | Wanderer unhealthy, 404: its web part calls its database by the public name, which a box without DNS cannot resolve | known: note on R-739 |
|
||||
| F3 | The install page's default place is the internal place `belso`, not the 210 GB drive | NOT-A-FINDING: by design — `16` §5 registers the internal place as the default drive |
|
||||
| F4 | Guide labels in English (LubeLogger „Settings", NZBGet „Settings / News-Servers") | NOT-A-FINDING: they are the app's own on-screen labels, and the Hungarian catalog copy is frozen by the copy-i18n gate (measured on the drill) |
|
||||
| F5 | Kimai opens in English | NOT-A-FINDING: a Hungarian browser lands on `/hu/`; only a request with no language gets `/en/` |
|
||||
| F6 | AdventureLog backend 95 % of its limit | NOT-A-FINDING: that was `docker stats` with the cache; on anon 288.7/384 MiB (75 %), = the ladder's 73 % |
|
||||
| — | Database error lines (SparkyFitness `database "sparky" does not exist`, Dawarich `database "dawarich" already exists`; SparkyFitness `Invalid password` = the harness's wrong-password control) | NOT-A-FINDING: the images' own first-start and restore-time checks; no restart, the item read back after the restore |
|
||||
| — | Radicale redirects to an IP | NOT-A-FINDING: a curl artefact (no Host on the redirect probe); the browser lands right |
|
||||
| — | NZBGet / Transmission screenshots show the browser's basic-auth prompt | NOT-A-FINDING: the app's own login, by design |
|
||||
| — | code-server's first install failed (`toomanyrequests`) | the failed install was removed through the product (`code-server.failed-remove.txt`); the re-run worked; the message is R-1273 |
|
||||
|
||||
## F10 proof — the kept unit is rewritten by the reinstall (`_unitprobe/tandoor/`)
|
||||
|
||||
`tools/unitprobe.py tandoor`: install, one account, „Mentés most" (unit 19:45:14–20), remove keeping data and
|
||||
backups, install again (a volume app: no kept-data question), wait for the controller's next capture, „Visszaállítás".
|
||||
|
||||
- 19:50:32 `Recovery unit captured for tandoor` — the unit's `app.yaml` checksum `0f366c97…` → `4438c1c5…`,
|
||||
`created_at` 19:45:26 → 19:50:32; the data files unchanged (`unit-1/2/3*.txt`).
|
||||
- The restore: `ok: true`, „A(z) tandoor: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult."
|
||||
- After: state `starting`, the account not there, `password authentication failed` ×42 (app) and ×22 (postgres).
|
||||
- The comment at `felhom-controller/controller/internal/backup/recovery_unit.go:101-103` says the refresh never rewrites
|
||||
the definition away from its data; after a remove + reinstall it does.
|
||||
|
||||
Seen three times: the sweep's own Tandoor cycle (17:27:41 reinstall → 17:30:34 capture → 17:30:47 restore), this probe,
|
||||
and AdventureLog's re-run (20:03:18 → 20:05:32 → restore). AdventureLog's first run worked only because no capture fell
|
||||
between its reinstall and its restore.
|
||||
|
||||
## D22 put-back — PASS on two apps (`_putback/`)
|
||||
|
||||
`tools/putback.py <app>`: the largest volume tar of the app's unit truncated to half, then „Visszaállítás". PrivateBin
|
||||
and Vikunja: the restore failed honestly („Nem változott semmi…"), the pre-restore data was put back, the app ran, and the
|
||||
seeded item read back through the front door.
|
||||
|
||||
## Drill fixes, proven on the sweep box pointed at the drill (Part 2)
|
||||
|
||||
Drill catalog `app-catalog-drill` (never the live catalog): `742a354` (merge of live main), then:
|
||||
|
||||
- `a601950` **n8n** — `N8N_DIAGNOSTICS_ENABLED=false` in the compose (usage data stays home).
|
||||
- `405077f` + `492a17f` **OpenGist** — memory 128M → 256M (compose + `.felhom.yml` `mem_limit`), the comment carries the
|
||||
measured need.
|
||||
|
||||
**How the box was pointed at the drill.** The drill repo on Gitea is private (R-992) and the box holds no catalog
|
||||
credential, so the box reads a local mirror: a bare repository in the controller's data volume, loaded from a git bundle,
|
||||
`repo_url: file:///opt/docker/felhom-controller/drill-catalog.git` (the method of
|
||||
`audits/burndown-3-2026-10-17/tools/repoint.py`; loading the mirror is a `docker exec git fetch` into the controller —
|
||||
catalog plumbing, not app state). Only `repo_url` changes; `tools/repoint.py live` puts the saved `controller.yaml` back
|
||||
and checks it byte for byte (`RESTORED-IDENTICAL`, twice in `_drill-repoint.txt`). The box ended on the live catalog
|
||||
`227a544`.
|
||||
|
||||
**What went wrong first, and is kept as evidence.** The first drill pass (chain 3, 21:18Z) set `repo_url` to the Gitea
|
||||
URL in the belief that the repo was public. The box's clone failed (`fatal: could not read Username`), the controller kept
|
||||
the templates it had, and n8n, OpenGist and Vaultwarden ran on the LIVE templates. The leftover `[Rudder]` lines in that
|
||||
n8n run showed it. Those three runs are kept as `run-live-2/` (valid live re-runs). The second pass (chain 4) refused to
|
||||
install anything until the box's catalog cache read the drill's commit and the changed lines were in the box's copy of
|
||||
both templates: `drill main fd23571, box cache fd23571: DRILL-CONFIRMED`. (The drill also carries a six-line Plex
|
||||
ladder entry from an earlier session's test; Plex was not installed in this pass.)
|
||||
|
||||
**n8n proof.** The running container carried `N8N_DIAGNOSTICS_ENABLED=false` (read at 21:37:58, that line only); error
|
||||
lines 0 against 6 `[Rudder]` lines in the live re-run just before it (the control); the owner account read back after the
|
||||
unit restore.
|
||||
|
||||
**OpenGist proof — and a wrong turn.** At 256M (drill) OpenGist took 215.8 MiB anon. I first read that as „it grows into
|
||||
its limit" and reverted the fix (`c420053`); an automated security review flagged the revert as a lowered resource cap.
|
||||
A measurement settled it (`tools/memprobe-guest.sh`, `opengist/memprobe-128M.txt`): on the live 128M template one
|
||||
sign-up + sign-in hit `memory.max` 443 times and pushed 98 MiB into swap (anon 125 MiB, 0 `oom_kill`, 0 restarts). The
|
||||
real need is ~220 MiB; the fix went back in (`405077f`). Where the box has no swap room left, the same sign-up would end in a kill (not measured here).
|
||||
|
||||
Drill gates: `catalog_gates.py --fast opengist n8n` from the drill clone — all OK except `security-scan`, which looks for
|
||||
a `felhom.eu` clone beside the drill and found none; the shared scanner run by hand on the drill clone: 0 hits.
|
||||
|
||||
## Teardown
|
||||
|
||||
Three layers, 2026-10-25 ~00:05Z:
|
||||
|
||||
- **Machine (demo-hp):** VM **9360** `sweep1024-box` and LXC **9361** `sweep1024-browser` — both created by this session —
|
||||
stopped and destroyed with `--purge`; no disk of either is left on `nvme-scratch`. Untouched: Tester 1 (VM 341), 9201,
|
||||
9202, 9401, the thin pool, and the 1.30.0 ISO in `local:iso` (there since 2026-10-22). Before the destroy, the box's
|
||||
controller log, the agent journal, its end state and its anonymous-volume list were copied here (`_box/`).
|
||||
- **Host (DooPlex):** the session's secret files (`/run/user/1000/sweep-sc`: the dashboard password copy, cookies, CSRF
|
||||
tokens) shredded. The drill catalog keeps its commits (`a601950`, `405077f`, `492a17f`, and the reverted turn
|
||||
`c420053`) for the morning list; nothing went to the live catalog.
|
||||
- **Hub:** host **Tester4-aa9466** stays for the operator to delete (CC holds no delete). Until then it reads as a box
|
||||
that stopped reporting.
|
||||
- **ep0:** listed, nothing removed. The host's PBS link to ep0 (`felhom-pbs`) was set up at day-0 (`pbsdr: converged`,
|
||||
14:44:48 CEST, `_box/agent-journal.log.gz`); no whole-box copy went to it (weekly cadence — only the local copy ran,
|
||||
14:51). The app copies the box sent off-site went to the Tester4 restic repository on the storage box (unlocked with
|
||||
the rehearsal's recovery code; a new code escrowed — stored out-of-band), not to ep0.
|
||||
|
||||
## Evidence map
|
||||
|
||||
- `_box/` — Part 0 (VM, ISO, bind, day-0, claim, recovery, drives, dashboard screens).
|
||||
- `<app>/` — one per app; re-run apps keep the earlier run in `first-run/`, `run-2/` or `run-drill-pre/`.
|
||||
- `<app>.console.txt`, `progress.log`, `batch.out`, `chain2.out`, `chain3.out` — the runner's own output.
|
||||
- `_unitprobe/tandoor/` — F10. `_putback/{privatebin,vikunja}/` — D22.
|
||||
- `tools/` — the harness (sweep copies of the catalog's `box_walk.py` and `upgrade_fixtures_box.py`, marked SWEEP COPY;
|
||||
`sweep.py`, `sweep_fixtures.py`, `unitprobe.py`, `putback.py`, `repoint.py`, `table.py`, the rehearsal's VM and browser
|
||||
helpers). Secrets were read by name through `sec` and never written here; the evidence was scanned for every known
|
||||
secret value and for token shapes (one app session token found and blanked: Actual Budget's), with a positive control.
|
||||
- Screenshots were reduced with `pngquant` (lossy palette) to keep the repository small.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Part 0 — the sweep box (2026-10-24, from 12:33Z)
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| VM | **9360 `sweep1024-box`** on demo-hp: 6 cores (`cpu host`), **12 GiB** (demo-hp had 14 GiB available of 29; 9201, 9202 and Tester 1 untouched — the brief's minimum, not its 16 GiB aim), 256 GB system disk + 200 GB + 150 GB data disks, all raw on `nvme-scratch` (the dir storage), never the thin pool (`vm-create.txt`) |
|
||||
| ISO | `felhom-installer-1.30.0-pve9.2-1.iso` — the current one on felhom.eu/letoltes; sha256 `590d7b9b…74d5cc1` = the published `.sha256` |
|
||||
| Installer | graphical, Hungarian keyboard kept, driven by `qm sendkey` (`tools/vmctl.sh`), screens in `install/`. Install pressed **12:37:33Z**; pairing code on the console **12:41** (blacked out in `install/11-console.png`) |
|
||||
| Bind | Tester4's old host deleted by the operator (12:32, asked by CC); the hub mailed a new bind link 12:31:57Z; bound through the real page in Chromium 12:42:18Z, first try (`screens/b-bind-*`) |
|
||||
| Day-0 | 8/8 steps, **SUCCESS 12:45:40Z** — host `Tester4-aa9466`, guest 9201, golden imported and verified; Install → done **8 min 7 s** |
|
||||
| Claim | setup-code mail 12:43:14Z („Új beállító kód — újratelepült a szervered", right for a customer who had a box before); claimed 302 → `/` (`claim.txt`) |
|
||||
| Recovery | the box found Tester4's sealed off-site package (2026-10-20 14:16) and asked for the old code first (`/recovery`): the 2026-10-20 code, typed as it was written down (ten words joined by hyphens), unlocked it — controller „the offsite repository key was recovered and placed" 12:47:41Z. Then the dashboard asked for a NEW code: the ceremony in Chromium (words 4 and 9 asked), escrowed; „Távoli mentés: Aktív" (`screens/box-escrow-*`, the code blacked out) |
|
||||
| Drives | `/dev/sdb` → `hdd_1` (210 GB, the apps' data), `/dev/sdc` → `hdd_2` (157 GB), both through `POST /api/storage/init` (`drive-init.txt`); the installer's internal place `belso` (59 GB) |
|
||||
| Versions | controller **0.342.0**, agent **0.164.0**, hub **0.169.0**; the box syncs the LIVE catalog (`app-catalog-felhom.eu` main, `5e196e1` at the start; it re-syncs every 15 min) |
|
||||
| Guest | 9201 at 192.168.0.157, 7.7 GiB RAM (the golden's size), domain `tester4.test` (no DNS — the browser LXC maps the names; TLS not verified) |
|
||||
| Browser | LXC **9361 `sweep1024-browser`** (2 cores, 2 GB, nvme-scratch): Chromium 156.0.8078.47 + Playwright 1.52.0; the household's session stored in the LXC (`/root/work/state-dash.json`) |
|
||||
| Dashboard | every menu page at phone and computer size, HU and EN (`screens-dashboard/`, 60): no sideways scroll reported; a signed-in household reads its OWN language setting whatever the browser asks — by design (`langFor`), the page texts were read with `?lang=` |
|
||||
|
||||
Secrets used (names only, `sec`): SWEEP_2026_10_24_ROOTPW (the typed root password — day-0 replaced it with the vaulted
|
||||
one; DooPlex's key was added before the bind), SWEEP_2026_10_24_BIND_TOKEN, _PAIRING_CODE, _SETUP_CODE, _DASH_PW,
|
||||
_RECOVERY_CODE, and the existing REHEARSAL_2026_10_20_PASSPHRASE / _RECOVERY_CODE.
|
||||
@@ -0,0 +1,52 @@
|
||||
2026-10-24T17:15:18Z 28 assets,config,library,
|
||||
2026-10-24T12:51:57Z 24 client_body,fastcgi,proxy,
|
||||
2026-10-24T13:08:14Z 24 client_body,fastcgi,proxy,
|
||||
2026-10-24T19:54:35Z 4
|
||||
2026-10-24T17:15:18Z 4
|
||||
2026-10-24T12:51:49Z 24 client_body,fastcgi,proxy,
|
||||
2026-10-24T19:54:18Z 4
|
||||
2026-10-24T13:06:43Z 24 client_body,fastcgi,proxy,
|
||||
2026-10-24T13:08:05Z 72 lock,nginx,nginx.pid,
|
||||
2026-10-24T19:06:36Z 36 documentserver,documentserver-example,
|
||||
2026-10-24T13:06:43Z 4
|
||||
2026-10-24T13:08:05Z 24 client_body,fastcgi,proxy,
|
||||
2026-10-24T13:06:43Z 72 lock,nginx,nginx.pid,
|
||||
2026-10-24T19:54:18Z 24 client_body,fastcgi,proxy,
|
||||
2026-10-24T17:18:02Z 4
|
||||
2026-10-24T13:07:11Z 24 client_body,fastcgi,proxy,
|
||||
2026-10-24T12:50:39Z 72 lock,nginx,nginx.pid,
|
||||
2026-10-24T13:07:11Z 72 lock,nginx,nginx.pid,
|
||||
2026-10-24T19:09:38Z 36 documentserver,documentserver-example,
|
||||
2026-10-24T17:20:41Z 28 assets,config,library,
|
||||
2026-10-24T19:54:35Z 24 client_body,fastcgi,proxy,
|
||||
2026-10-24T12:50:39Z 24 client_body,fastcgi,proxy,
|
||||
2026-10-24T12:51:57Z 72 lock,nginx,nginx.pid,
|
||||
2026-10-24T19:07:30Z 4
|
||||
2026-10-24T12:50:57Z 24 client_body,fastcgi,proxy,
|
||||
2026-10-24T19:21:09Z 4
|
||||
2026-10-24T19:09:38Z 4
|
||||
2026-10-24T12:50:57Z 72 lock,nginx,nginx.pid,
|
||||
2026-10-24T19:20:01Z 4
|
||||
2026-10-24T19:55:03Z 4
|
||||
2026-10-24T12:51:57Z 4
|
||||
2026-10-24T19:54:18Z 72 lock,nginx,nginx.pid,
|
||||
2026-10-24T17:18:02Z 28 assets,config,library,
|
||||
2026-10-24T19:24:43Z 4
|
||||
2026-10-24T12:50:39Z 4
|
||||
2026-10-24T13:08:05Z 4
|
||||
2026-10-24T12:51:49Z 4
|
||||
2026-10-24T19:07:30Z 32 documentserver,documentserver-example,
|
||||
2026-10-24T19:06:36Z 4
|
||||
2026-10-24T19:07:39Z 4
|
||||
2026-10-24T13:08:14Z 4
|
||||
2026-10-24T17:17:24Z 4
|
||||
2026-10-24T12:50:57Z 4
|
||||
2026-10-24T17:17:24Z 28 assets,config,library,
|
||||
2026-10-24T19:55:03Z 24 client_body,fastcgi,proxy,
|
||||
2026-10-24T19:55:03Z 72 lock,nginx,nginx.pid,
|
||||
2026-10-24T13:08:14Z 72 lock,nginx,nginx.pid,
|
||||
2026-10-24T19:07:39Z 36 documentserver,documentserver-example,
|
||||
2026-10-24T19:54:35Z 72 lock,nginx,nginx.pid,
|
||||
2026-10-24T17:20:41Z 4
|
||||
2026-10-24T12:51:49Z 72 lock,nginx,nginx.pid,
|
||||
2026-10-24T13:07:11Z 4
|
||||
@@ -0,0 +1,3 @@
|
||||
GET /claim HTTP/2 200
|
||||
csrf_len 64 cookie_present yes
|
||||
POST /claim HTTP/2 302 location: /
|
||||
@@ -0,0 +1,11 @@
|
||||
candidates 200 {"data": {"vmid": 9201, "initialize": [{"device": "/dev/sdb", "size_bytes": 214748364800, "model": "QEMU HARDDISK", "data_bearing": false, "mountable": false}, {"device": "/dev/sdc", "size_bytes": 161061273600, "model": "QEMU HARDDISK", "data_bearing": false, "mountable": false}], "attach": []}, "ok": true}
|
||||
12:50:04Z init /dev/sdb hdd_1 200 {"data": {"phase": "formatting", "started": true}, "ok": true}
|
||||
status 200 {"device": "/dev/sdb", "durable_id": "", "error": "", "opsign": "", "phase": "done", "reason": "", "started_at": "2026-10-24T12:50:04.496400956Z", "updated_at": "2026-10-24T12:50:06.691612961Z", "where": "/mnt/felhom-drives/hdd_1"}
|
||||
12:50:09Z init /dev/sdc hdd_2 200 {"data": {"phase": "formatting", "started": true}, "ok": true}
|
||||
status 200 {"device": "/dev/sdc", "durable_id": "", "error": "", "opsign": "", "phase": "done", "reason": "", "started_at": "2026-10-24T12:50:09.550638661Z", "updated_at": "2026-10-24T12:50:11.762283786Z", "where": "/mnt/felhom-drives/hdd_2"}
|
||||
9dcbe7bf-098d-4d99-9c43-f39fe2b20f42 disk user-data attached 157 GB /mnt/felhom-drives/hdd_2
|
||||
belso internal user-data attached 59 GB /mnt/felhom-drives/belso
|
||||
e8b4f4a0-7aca-4829-9220-b394948b9368 disk user-data attached 210 GB /mnt/felhom-drives/hdd_1
|
||||
local system system attached 77 GB None backup_target
|
||||
local-lvm system system attached 165 GB None
|
||||
felhom-pbs network backup attached 0 GB None
|
||||
@@ -0,0 +1,8 @@
|
||||
filebrowser gtstef/filebrowser:1.5.6-stable Up 9 hours (healthy)
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 6 minutes (healthy)
|
||||
traefik traefik:v3.7.13 Up 9 hours
|
||||
54
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/sdb 196G 268M 186G 1% /mnt/felhom-drives/hdd_1
|
||||
/dev/sdc 147G 2.2M 140G 1% /mnt/felhom-drives/hdd_2
|
||||
/dev/mapper/pve-vm--9201--disk--0 32G 1.2G 29G 4% /
|
||||
|
After Width: | Height: | Size: 58 KiB |
|
After Width: | Height: | Size: 58 KiB |
|
After Width: | Height: | Size: 57 KiB |
|
After Width: | Height: | Size: 61 KiB |
|
After Width: | Height: | Size: 60 KiB |
|
After Width: | Height: | Size: 67 KiB |
|
After Width: | Height: | Size: 55 KiB |
|
After Width: | Height: | Size: 60 KiB |
|
After Width: | Height: | Size: 60 KiB |
|
After Width: | Height: | Size: 55 KiB |
|
After Width: | Height: | Size: 3.9 KiB |
@@ -0,0 +1 @@
|
||||
Install pressed 12:37:33
|
||||
|
After Width: | Height: | Size: 28 KiB |
|
After Width: | Height: | Size: 30 KiB |
|
After Width: | Height: | Size: 46 KiB |
|
After Width: | Height: | Size: 42 KiB |
|
After Width: | Height: | Size: 65 KiB |
|
After Width: | Height: | Size: 65 KiB |
|
After Width: | Height: | Size: 141 KiB |
|
After Width: | Height: | Size: 141 KiB |
|
After Width: | Height: | Size: 29 KiB |
|
After Width: | Height: | Size: 29 KiB |
|
After Width: | Height: | Size: 46 KiB |
|
After Width: | Height: | Size: 44 KiB |
|
After Width: | Height: | Size: 50 KiB |