Night 2026-10-25 Part 4–5: each fix's [DESIGN] line (04, 05, 07, 08, 11, 13, 15), the register (R-1261, R-1281, R-1282, R-1288, R-1280, R-1285, R-1283, R-1287, R-1289, R-1278 to VERIFY; R-1286 waiting on the operator; R-1291 and R-1292 opened), STATUS (D51 Kimai, the Monday list), the night report with its three-layer teardown, and the rig evidence (scanned: 0 known-secret hits)
@@ -1720,7 +1720,7 @@ names the app and both identities). Nothing moves a unit while that app is being
telepítés mentése") with **„Visszaállítás"** — the one unit-restore body, with the unit's own database password — and
„Eltávolítás". Only those two, and the app's remove with „delete its backups too", ever remove it (the §8.2a / R-1252 H9
rule). After a restore from it, the app is that unit's install; the app's own unit of the install it replaced is then
set aside at the next backup — never lost.
set aside ~~at the next backup~~**at the restore itself** — never lost. **[DESIGN — R-1288, 2026-10-25 night, the brief's safest reading; controller `b5863a9`, unreleased]:** the restore of a set-aside unit moves the replaced install's unit aside whole right after it (success, or „not started yet" — the data stays, D49 = A), one log line „replaced by the restored install", **no operator alarm** — the household chose this restore; before, the next backup's writer set it aside with a second alarm and a second kept copy (measured on Tester 1, 0.343.0). **[FACT]** on the night's rig (rc.2): the line, the account back, the next backup set nothing aside, one set-aside mail only (`audits/night-2026-10-25/rig/r1288-live.txt`). A set-aside's operator mail (R-1287, hub v0.171.0, unreleased) reads „⚠️ … a backup of an earlier install was set aside — nothing failed", a warning.
**Known limit, stated.** A box that reinstalled and refreshed BEFORE this release already holds a rewritten unit; its
old password is gone and nothing can bring it back. A unit from before D47 of an app with no carried secret has no
@@ -1728,7 +1728,7 @@ signal: it is treated as this install's (the old behaviour) and gets the identit
**Why a volume app's reinstall asks no kept-data question:** the question (§6.5, decision 36) is about the app's drive
FOLDER (`${HDD_PATH}` binds under `appdata/`); a volume app keeps nothing there — „remove, keep my data" removes its named
volumes and keeps only its backups — so the install starts empty, and the kept backup is now the set-aside unit on the
restore page. (The missing offer at install is its own row, R-1278.)
restore page. **[DESIGN — R-1278, controller `fbec5cb`, unreleased]:** the install page now says so for an app not installed — „the new install starts empty; load the backup afterwards on the Restore page" — with a link (set-aside and removed-app units counted).
### 8.2e A restore whose app does not start is never „success" [DESIGN, R-1270, 2026-10-25 — controller `b680db8` + fix A; D49 = A (operator 2026-10-25); released in controller 0.343.0]
@@ -591,6 +591,15 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me
start-up check puts it back; the 50 % rule waits while a record exists. Measured on demo-hp with a real seed (10
Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; with it the uplink fell to 0.02–0.16 Mbit/s,
94.2 Mbit/s measured, the app held 56.8 s and back to 1 249 280 B/s exactly (`audits/dooplex-alarms-2026-10-24/`).
**[DESIGN — R-1261, 2026-10-25 night; controller `a79a292`, unreleased]:** EVERY running seeder is held (qBittorrent
and Transmission — one left seeding kept the test refused `not-quiet` every time); all records first
(`upwatch.line_test_holds`), then the apps; a record whose app was removed while held is dropped (it used to stop the
50 % rule for good); qBittorrent's hold names its slot (`up_limit` / `alt_up_limit` through `app/preferences` —
`transfer/uploadLimit` follows the CURRENT speed mode, measured on 5.2.4), so the household's switch to the
alternative speed during the test never leaves 1 KiB/s behind. **[FACT]** the night's rig (BYO VM, agent 0.165.0,
rc.1, two seeders to 9202's qBittorrent): with a mode switch mid-test both limits came back, the test refused
(the switch let qBittorrent upload); without, both held (qBittorrent in its alternative slot), 94.8 Mbit/s measured,
both limits back exactly (`audits/night-2026-10-25/rig/hold-watch*.txt`).
While the box owns it, qBittorrent's page says: „A qBittorrent
feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
6. **Limits, stated:***(added 2026-10-24, review round 2: ANY viewer explains any amount of upload, so a hacked app can send freely while one real playback runs — `upwatch/watch.go:275,299`; R-1262)*; an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
@@ -169,8 +169,10 @@ download (R-1019). *(Fixed since: catalog `c50adfc` marks the download and watch
`app-catalog-felhom.eu/templates/transmission/.felhom.yml:46-51`; noted 2026-10-24.)* **[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1283]:** the same commit also
marks the download folders of NZBGet, SABnzbd and MeTube `excluded` (`templates/{nzbget,sabnzbd,metube}/.felhom.yml`).
NZBGet and SABnzbd stay excluded (their files can be fetched again). The pick for MeTube: `optional`, so the
second-drive copy carries it and the off-site copy does not — a MeTube video can be gone from its source site. Not
built yet. **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
second-drive copy carries it and the off-site copy does not — a MeTube video can be gone from its source site.
**Built in the DRILL catalog 2026-10-25 night (`app-catalog-drill``6733764`, not live):** `class: optional`, and the
download-binds gate allows `optional` only for an app named with its ruling (`OPTIONAL_BY_RULING`); mandatory and
unlisted stay refused. The live push is the operator's (the Monday list). **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
media is no single app's data; whether the library gets its own household-level backup is a separate decision.
## 8. Removal and export assume one drive per app
@@ -275,7 +277,9 @@ returns** (G7f).
**[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1282]:** the „Később"
03:30 media restart is an IN-MEMORY timer — a controller restart forgets it and only the notice stays
(`internal/web/media_events.go``handleMediaRestart`, `time.AfterFunc`). The pick: the controller stores the due
time and runs the restart after a restart of its own. Not built yet.
time and runs the restart after a restart of its own. **Built 2026-10-25 night (controller `b5863a9`, unreleased):**
`settings.media_restart_later_at`; at start `ResumeMediaLater` re-arms it, runs it at once when it passed while the
controller was down, and clears it once it ran.
**[DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]:** the Sonarr/Radarr API key is READ from
`/config/config.xml`; the „set at install" option §6 calls cleaner was not built (`internal/mediawire/mediawire.go:336-345`).
=== RUN TestR1281_OnlyOfficeJoinsTheFilesNetworkOnlyWhileLinked
filebrowser_password_test.go:385: a linked OnlyOffice must reach FileBrowser: map[3f2a9c@felhom-files:true felhom-controller@felhom-filebrowser:true traefik@felhom-files:true]
=== RUN TestR1285_AnInstalledBundleIsFollowedByThePolicyPass
bundle_test.go:173: after the install the agent must run the policy pass (host layer, no envelope): [map[bundle:/tmp/TestR1285_AnInstalledBundleIsFollowedByThePolicyPass3671177433/001/bundle-0.143.0.json layer:host mode:bundle release_id:bundle-0.143.0 signed:map[blob_b64:eyJvcCI6ImFnZW50X2NvbmZpZ191cGRhdGUifQ== sig:SIG]]]
# R-1286 red-proof: holdWhy back to the old two-way branch, 2026-10-25 night
=== RUN TestR1286_EveryHoldReasonSaysWhatItIs
r1286_hold_why_test.go:32: kimai (reason "unhealthy_stop"): the gate says "held after a failed restore whose rollback also failed (2026-10-14T09:06:31Z) — clear the hold to start it", want it to contain "stopped by the box for a crash loop"
=== RUN TestR1282_AFutureLaterRestartIsKeptOverAStart
r1282_media_later_test.go:74: the timer must be armed for 2026-10-25 22:29:25.783026813 +0100 CET m=+3602.004971479, got 0001-01-01 00:00:00 +0000 UTC (armed false)
2026/10/25 20:28:03 [INFO] [upwatch] line test: qbittorrent held at {Bytes:1024 Alt:true Enabled:false KBps:0} (was {Bytes:3072000 Alt:true Enabled:false KBps:0})
2026/10/25 20:28:03 [INFO] [upwatch] line test: transmission held at {Bytes:0 Alt:false Enabled:true KBps:1} (was {Bytes:0 Alt:false Enabled:false KBps:500})
2026/10/25 20:28:03 [INFO] [upwatch] line test: 2 app(s) held — asking the agent for a test
2026/10/25 20:29:00 [INFO] [upwatch] qbittorrent's upload limit is back as it was ({Bytes:3072000 Alt:true Enabled:false KBps:0})
2026/10/25 20:29:00 [INFO] [upwatch] transmission's upload limit is back as it was ({Bytes:0 Alt:false Enabled:false KBps:500})
2026/10/25 20:29:00 [INFO] [upwatch] line test on request: 94.8 Mbit/s
2026/10/25 20:29:00 [INFO] [upwatch] the host's weekly line test: 94.8 Mbit/s (measured 2026-10-25T20:28:18Z)
== agent lines
Oct 25 12:41:54 pf-test felhom-agent[1117]: time=2026-10-25T20:41:54.210+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backu
Oct 25 12:48:11 pf-test felhom-agent[11486]: time=2026-10-25T12:48:11.728-07:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a back
Oct 25 13:13:03 pf-test felhom-agent[11486]: time=2026-10-25T13:13:03.535-07:00 level=INFO msg="local-api: line test requested" vmid=9201
Oct 25 13:13:18 pf-test felhom-agent[11486]: time=2026-10-25T13:13:18.538-07:00 level=INFO msg="linetest: test on request refused — the uplink is not quiet" mbps=10.0
Oct 25 13:13:18 pf-test felhom-agent[11486]: time=2026-10-25T13:13:18.539-07:00 level=INFO msg="local-api: line test request refused" vmid=9201 reason=not-quiet detail="the uplink sent 10.0 Mbit/s while held"
Oct 25 13:28:03 pf-test felhom-agent[11486]: time=2026-10-25T13:28:03.644-07:00 level=INFO msg="local-api: line test requested" vmid=9201
Oct 25 13:28:11 pf-test felhom-agent[11486]: time=2026-10-25T13:28:11.729-07:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=line-test
Oct 25 13:28:18 pf-test felhom-agent[11486]: time=2026-10-25T13:28:18.647-07:00 level=INFO msg="linetest: test on request (a held torrent app, R-1250)" server=10.77.0.1:5201
Oct 25 13:28:29 pf-test felhom-agent[11486]: time=2026-10-25T13:28:29.021-07:00 level=INFO msg="linetest: run 1: 94.8 Mbit/s" server=10.77.0.1:5201 server_bytes=119144448 server_seconds=10.058812
Oct 25 13:28:44 pf-test felhom-agent[11486]: time=2026-10-25T13:28:44.399-07:00 level=INFO msg="linetest: run 2: 94.7 Mbit/s" server=10.77.0.1:5201 server_bytes=119013376 server_seconds=10.057222
Oct 25 13:28:59 pf-test felhom-agent[11486]: time=2026-10-25T13:28:59.786-07:00 level=INFO msg="linetest: run 3: 94.4 Mbit/s" server=10.77.0.1:5201 server_bytes=118751232 server_seconds=10.067068
Oct 25 13:28:59 pf-test felhom-agent[11486]: time=2026-10-25T13:28:59.786-07:00 level=INFO msg="linetest: upload line measured: 94.8 Mbit/s (best of 3)" server=10.77.0.1:5201 runs=3 duration_s=41.138682127
2026/10/25 19:58:03 [INFO] [upwatch] qBittorrent's upload limit is the household's — the box leaves it alone from now on
2026/10/25 20:13:03 [INFO] [upwatch] line test: qbittorrent held at {Bytes:1024 Alt:false Enabled:false KBps:0} (was {Bytes:2499584 Alt:false Enabled:false KBps:0})
2026/10/25 20:13:03 [INFO] [upwatch] line test: transmission held at {Bytes:0 Alt:false Enabled:true KBps:1} (was {Bytes:0 Alt:false Enabled:false KBps:500})
2026/10/25 20:13:03 [INFO] [upwatch] line test: 2 app(s) held — asking the agent for a test
2026/10/25 20:13:18 [INFO] [upwatch] qbittorrent's upload limit is back as it was ({Bytes:2499584 Alt:false Enabled:false KBps:0})
2026/10/25 20:13:19 [INFO] [upwatch] transmission's upload limit is back as it was ({Bytes:0 Alt:false Enabled:false KBps:500})
2026/10/25 20:13:19 [INFO] [upwatch] line test on request refused by the agent: not-quiet (the uplink sent 10.0 Mbit/s while held)
== agent lines
Oct 25 12:25:22 pf-test felhom-agent[2092]: time=2026-10-25T20:25:22.507+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backu
Oct 25 12:41:54 pf-test felhom-agent[1117]: time=2026-10-25T20:41:54.210+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backu
Oct 25 12:48:11 pf-test felhom-agent[11486]: time=2026-10-25T12:48:11.728-07:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a back
Oct 25 13:13:03 pf-test felhom-agent[11486]: time=2026-10-25T13:13:03.535-07:00 level=INFO msg="local-api: line test requested" vmid=9201
Oct 25 13:13:18 pf-test felhom-agent[11486]: time=2026-10-25T13:13:18.538-07:00 level=INFO msg="linetest: test on request refused — the uplink is not quiet" mbps=10.0
Oct 25 13:13:18 pf-test felhom-agent[11486]: time=2026-10-25T13:13:18.539-07:00 level=INFO msg="local-api: line test request refused" vmid=9201 reason=not-quiet detail="the uplink sent 10.0 Mbit/s while held"
# off-site half: the rig has no off-site backup set up — not run live; governed by appbackup.tierKeeps (optional → TierSecondary only, internal/appbackup/captureset.go:191-199)
# R-1285 live on the rig (BYO host Tester4-84bbe7), 2026-10-25 night
# staged: new agent 0.166.0-r1285 (main abcc1e6) installed by hand; the OLD 0.164.0 wrapper put back (no bundle-policy mode); our 0.164.0 drop-in planted (sha256 f0485da5…, sshd -T without-password)
# then ONE signed agent_config_update, test bundle 0.166.0-r1285 (sha256 e73bdb6c…)
Oct 25 14:53:48 pf-test felhom-agent[137016]: time=2026-10-25T14:53:48.913-07:00 level=WARN msg="signedjobs: signed op COMPLETED" job=14de31ae946b9f76 op=agent_config_update
20:52:08Z tandoor: readback of the seeded account found=True
20:52:08Z item back: True
20:52:08Z controller lines since the restore:
2026/10/25 20:51:33 [INFO] [backup] tandoor: the unit of the install the restore replaced (7dc433bf4851beb7) is set aside whole → /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T205133Z — replaced by the restored install e30f6554375a455b; no
- the escrow blob in the hub, if one exists (operator UI)
[0;32m[OK][0m UNINSTALL complete — removed: guest 9201, the felhom-agent (unit/sudoers/binary/state/config+baks/user + selfupdate-artifacts/shared-parent/mkfs-wrapper/pbs-apply-wrapper/hook-snippet/dnsmasq-snippets), the pveum role/user/token/ACL, the felhom pool, and /var/lib/felhom-install/state.json.
[0;32m[INFO][0m golden vzdump: left in place (--remove-golden to remove).
[0;32m[INFO][0m NOTE: the 'sudo' package was left INSTALLED (a system package, never ours to remove). dnsmasq is removed only when the install-time record says Felhom installed it; a resolver that pre-dated Felhom is never touched. The host record still exists in the hub — remove it there if desired.
say("set-aside dirs after the restore:",sw.g(f"find /mnt/felhom-drives -path '*set-aside/tandoor/*' -maxdepth 6 -mindepth 6 -type d 2>/dev/null | wc -l").strip())
t_b=int(time.time())
sw.backup_now({},say)
say("after the next backup — set-aside lines:",sw.g(f"docker logs --since {t_b} felhom-controller 2>&1 | grep -c 'recovery unit of an earlier install'").strip(),
| **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).***Original finding, kept:***[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.****What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC****Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC |
| **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)****Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator |
## Apps & catalog — 16 rows (P3 6, P4 10)
## Apps & catalog — 17 rows (P3 6, P4 11)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -145,6 +145,7 @@ stopping line that lies.
| **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | WATCHING — **2026-10-22 night:** not reproducible tonight; the Immich server log of 19:38 is gone (Immich restarted at 02:16 for the off-site dump). What the household sees: the upload fails after 10 minutes and the Immich phone app sends it again. | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC |
| **R-1267** | Apps & catalog | P4 | **Every recreate of Immich leaves one empty anonymous Docker volume behind: the image declares `VOLUME /data`, the template does not map it.** Found 2026-10-24 (review round 2, while putting 9202 back): 9202 holds 296 empty anonymous volumes; each night chain's recreate of `immich-server` made a new one (12:58, 13:00, 13:01 — the three test runs; the two dangling ones this session made were removed by name, nothing pruned). The photos are NOT affected: Immich picks `/usr/src/app/upload` whenever both folders exist (`detectMediaLocation` in the image's `server/dist/services/storage.service.js:28-39`), and the template binds that folder (`templates/immich/docker-compose.yml:49`) — NOT-A-FINDING for data placement. **Seen again 2026-10-24 night on Tester 1:** privatebin's image declares `/run`, `/tmp` and `/var/lib/nginx/tmp` as volumes — each recreate (the night dump's stop/start) leaves three empty anonymous volumes; the same class as Immich's `/data` (`audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt`). | OPEN **2026-10-24 (catalog sweep):** the same class for OnlyOffice and RomM (RomM's image volume `/romm` holds only the empty mount points of its binds); after every catalog app had been removed WITH data, the sweep box still held 52 such volumes (PrivateBin 21, OnlyOffice 8, RomM 8, Immich 3, and 12 from 12:50–12:52 before the batch, holding the same nginx folders as PrivateBin's) — remove-with-data does not see them (no compose project label). All small (4–72 KB), no household data (`audits/catalog-sweep-2026-10-24/_box/anon-volumes.txt`). | — | declare the image's VOLUME paths as tmpfs or named volumes in the catalog (Immich, privatebin), or remove the dangling ones a recreate leaves, by name | CC |
| **R-1268** | Backup & restore | P3 | **Remove with „remove backups” leaves a restore's undo copy on the drive (`backups/restore-undo/<app>/`).** Seen 2026-10-24 night on Tester 1 (restore-holes Part 1, run 1): MeTube was held for support after a failed crash put-back, then removed with `remove_backups=true` — the 1.5 GB copy stayed (the removal deletes `backups/primary` and the second-drive mirror, not `restore-undo`), and nothing lists it (`audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt`; removed by hand, CC's own test data). The copy is kept on purpose while a hold exists; after the app is gone it is orphaned. | OPEN | — | Remove with `remove_backups` also deletes the app's `restore-undo` folder (under the same path guard), or lists it as a kept copy | CC |
| **R-1292** | Apps & catalog | P4 | **qBittorrent's 50 % rule reads and writes the limit of the CURRENT speed mode: with the household's alternative speed on, it sets the alternative limit, and a later switch reads as „the household changed it" — the rule stops for good.** Measured 2026-10-25 night on qBittorrent 5.2.4 (`transfer/uploadLimit`/`setUploadLimit` follow the mode; `audits/night-2026-10-25/r1261-qbittorrent-api-measured.txt`); found while fixing R-1261 (the hold now names its slot). | OPEN | — | make the rule own the NORMAL slot only (`up_limit` through `app/preferences`, as the hold does) + a test with the alternative mode on (red first); one controller release | CC |
## App updates — 5 rows (P2 1, P3 4)
@@ -176,7 +177,7 @@ stopping line that lies.
| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. | OPEN | — | one attended pass over the restore paths with these four as tests | CC |
| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
| **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
| **R-1278** | Backup & restore | P3 | **A volume app's reinstall does not offer its kept backup.** „Remove — keep my data” removes a volume app's named volumes and keeps only its backup; the kept-data question is about drive folders only (`07` §6.5, decision 36), so the reinstall starts empty and says nothing. Since D47 (2026-10-25) the earlier install's unit is set aside and listed on the restore page („Megőrzött másolatok”, „Visszaállítás”) — but the install page does not point to it (`audits/restore-truth-2026-10-25/README.md`). | OPEN | — | Design: at install, when a set-aside or removed-app unit of the app exists, say so and offer it beside a fresh start; a test | CC |
| **R-1278** | Backup & restore | P3 | **A volume app's reinstall does not offer its kept backup.** „Remove — keep my data” removes a volume app's named volumes and keeps only its backup; the kept-data question is about drive folders only (`07` §6.5, decision 36), so the reinstall starts empty and says nothing. Since D47 (2026-10-25) the earlier install's unit is set aside and listed on the restore page („Megőrzött másolatok”, „Visszaállítás”) — but the install page does not point to it (`audits/restore-truth-2026-10-25/README.md`). | **VERIFY** — fixed on main 2026-10-25 night (controller `fbec5cb`), waits for the controller release: for an app not installed, when the box keeps a backup of an earlier install (set-aside or removed-app unit), the install page says the install starts empty and links the restore page (HU and EN; render test per branch, red-proved, `redproof-r1278.txt`). Not seen live. | — | Design: at install, when a set-aside or removed-app unit of the app exists, say so and offer it beside a fresh start; a test | CC |
| **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
| **R-164** | Backup & restore | P4 | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:**`ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
| **R-213** | Backup & restore | P4 | **Putting files back in place — the half the recovery screen deliberately does not do.** The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: *a screen that unlocks and then offers to overwrite is two decisions wearing one button*. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. **The operator named its requirement: a live-versus-backup comparison** — the customer must be able to see what would change before anything is overwritten | **DEFERRED** — operator ruling 2026-10-12 (S18, `09` §3 decision 213): designed after the first customers. Stays P4; re-dated: look again when the first paying household has run 30 days. **OPEN — not started, deliberately** | the comparison design (nothing exists for it yet) | After the first customers | Operator + CC |
@@ -212,9 +213,9 @@ stopping line that lies.
| **R-1232** | Backup & restore | P2 | **A reinstalled box may take its first whole-box copy to ep0 up to a week late: the old box's copy in the same household space seems to count as its last one.** Test night 2026-10-23 (`audits/night-2026-10-23/P4-testnight/`): the new Tester4 box (guest 9201) had its PBS tier applied at 19:28 UTC, but no whole-box run to ep0 came all night; the only archive in the household's ep0 space is the rehearsal box's `ct/9201` of 2026-10-20 (same guest number), and the tier's cadence is 168 h. The agent's due-check reads the newest archive for that vmid on the tier's storage (`localapi/server.go``newestArchiveOn`) — so the old box's copy reads as „2.6 days old, not due”. If so, the ep0 copy is the OLD box's for up to a week. PLAUSIBLE — read from the code and the night's silence, not proven by a log line (the controller logs the due-check only at debug). | OPEN | — | Prove: read `GET /backup/tiers` for the box (the agent's answer per tier) on the next reinstall; fix: a tier's „last backup” counts only archives this host wrote (or after the host's enrollment); an agent test with a foreign archive in the namespace | CC |
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
| **R-1283** | Backup & restore | P3 | **MeTube's downloads are excluded from every backup — make them `optional` (D44 #25).** Catalog `c50adfc` (R-1019) marks the download folders of qBittorrent, Transmission, NZBGet, SABnzbd and MeTube `excluded` (`app-catalog-felhom.eu/templates/metube/.felhom.yml:33`). Risk: a MeTube video may be gone from its source site, so after a drive failure it cannot be fetched again. Pick: MeTube's `media/metube` becomes `optional` — the second-drive copy carries it, the off-site copy does not; NZBGet and SABnzbd stay excluded (re-downloadable). Cost: second-drive space for MeTube's folder. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §7. | OPEN | — | change the class in the catalog template, run the catalog gates (the download-binds gate and its decoys must accept MeTube as `optional`), check on a Tier-0 box that a second-drive copy carries the folder and the off-site run does not | CC |
| **R-1286** | Backup & restore | P4 | **demo-hp's kimai has been HELD since 2026-10-14 09:06Z („held after a failed restore whose rollback also failed") and no row names it.** Seen 2026-10-25 by the D47 upgrade count (attended release run): kimai is the one app of 12 on demo-hp without an install id, because the start-up step skips a held app; `app.yaml` says deployed, no container exists, boot reconciliation leaves it held, the night update leg has skipped it since at least 2026-10-16 (`audits/burndown-apps-2026-10-16/P1/night-chain-log.txt`). A test box; the hold is the designed fail-closed state (R-379), but 11 days with no record is how a held household app would go unseen too. | OPEN | — | read why the 2026-10-14 restore and its rollback failed (demo-hp's controller log of that day, if kept; the hold record); then clear the hold (`--clear-restore-hold kimai` + a restart) or remove the app through the product; check that a held app raises a standing operator signal, else file that | CC |
| **R-1288** | Backup & restore | P4 | **After the household restores an earlier install's set-aside copy, the next backup sets the REINSTALL's own unit aside too — a second kept copy on the restore page and a second operator alarm.** Seen 2026-10-25 on Tester 1 (0.343.0, Tandoor, attended release run): reinstall → its first backup set install A's unit aside (17:31:55) and wrote install B's unit; restoring A's copy makes the app install A again („the restored data belongs to install 795457b5… — its backups follow it (D47)"); the next 20-minute backup step (17:36:54) found B's unit in `primary/` and set it aside. Nothing is lost; the second copy is the fresh reinstall's near-empty data. D47's design does what it says; the question is whether a restore of a set-aside unit should retire the unit it replaces instead of leaving it for the next writer. | OPEN | — | read `07` §8.2d/§8.2e for the intended end state; if the restore should retire B's unit: move it aside AT the restore (one line, no alarm, „replaced by the restored install") with a test; else document the second copy and word its alarm (R-1287) | CC |
| **R-1283** | Backup & restore | P3 | **MeTube's downloads are excluded from every backup — make them `optional` (D44 #25).** Catalog `c50adfc` (R-1019) marks the download folders of qBittorrent, Transmission, NZBGet, SABnzbd and MeTube `excluded` (`app-catalog-felhom.eu/templates/metube/.felhom.yml:33`). Risk: a MeTube video may be gone from its source site, so after a drive failure it cannot be fetched again. Pick: MeTube's `media/metube` becomes `optional` — the second-drive copy carries it, the off-site copy does not; NZBGet and SABnzbd stay excluded (re-downloadable). Cost: second-drive space for MeTube's folder. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §7. |**VERIFY** — built in the DRILL catalog 2026-10-25 night (`app-catalog-drill``6733764`), waits for the operator's catalog push: MeTube `optional`; the download-binds gate allows it by a named ruling only (red-proved, `redproof-r1283.txt`). **Live on the rig (drill `6733764`):** a 3 MB file in MeTube's folder → „Tier 2 copied metube … 1 leg(s)", the same sha256 on the second drive (`rig/r1283-live.txt`). The off-site half NOT live (the rig has no off-site): it is `appbackup.tierKeeps` (optional → the second drive only). **Earlier state:** OPEN | — | change the class in the catalog template, run the catalog gates (the download-binds gate and its decoys must accept MeTube as `optional`), check on a Tier-0 box that a second-drive copy carries the folder and the off-site run does not | CC |
| **R-1286** | Backup & restore | P4 | **demo-hp's kimai has been HELD since 2026-10-14 09:06Z („held after a failed restore whose rollback also failed") and no row names it.** Seen 2026-10-25 by the D47 upgrade count (attended release run): kimai is the one app of 12 on demo-hp without an install id, because the start-up step skips a held app; `app.yaml` says deployed, no container exists, boot reconciliation leaves it held, the night update leg has skipped it since at least 2026-10-16 (`audits/burndown-apps-2026-10-16/P1/night-chain-log.txt`). A test box; the hold is the designed fail-closed state (R-379), but 11 days with no record is how a held household app would go unseen too. |**WAITING-ON-OPERATOR** — read 2026-10-25 night: the hold is NOT a failed restore — `reason: unhealthy_stop`, `crash_loop`, 2026-10-14 09:06:31Z, during the night power drill on demo-hp (8 real cuts); every surface misnamed it — fixed on main (controller `838c834`, `holdWhy`, red-proved). A held app gives the operator no lasting signal → R-1291. **The pick (STATUS):** press Start (the household's Start lifts this hold; a new crash loop holds it again) or remove Kimai through the product. **Earlier state:** OPEN | — | read why the 2026-10-14 restore and its rollback failed (demo-hp's controller log of that day, if kept; the hold record); then clear the hold (`--clear-restore-hold kimai` + a restart) or remove the app through the product; check that a held app raises a standing operator signal, else file that | CC |
| **R-1288** | Backup & restore | P4 | **After the household restores an earlier install's set-aside copy, the next backup sets the REINSTALL's own unit aside too — a second kept copy on the restore page and a second operator alarm.** Seen 2026-10-25 on Tester 1 (0.343.0, Tandoor, attended release run): reinstall → its first backup set install A's unit aside (17:31:55) and wrote install B's unit; restoring A's copy makes the app install A again („the restored data belongs to install 795457b5… — its backups follow it (D47)"); the next 20-minute backup step (17:36:54) found B's unit in `primary/` and set it aside. Nothing is lost; the second copy is the fresh reinstall's near-empty data. D47's design does what it says; the question is whether a restore of a set-aside unit should retire the unit it replaces instead of leaving it for the next writer. |**VERIFY** — fixed on main 2026-10-25 night (controller `b5863a9`; the brief's safest reading, `07` §8.2d), waits for the controller release: the replaced install's unit set aside AT the restore, no alarm. **Live on the rig (rc.2):** the line, the account back, the next backup set nothing aside, one set-aside mail only (`rig/r1288-live.txt`). **Earlier state:** OPEN | — | read `07` §8.2d/§8.2e for the intended end state; if the restore should retire B's unit: move it aside AT the restore (one line, no alarm, „replaced by the restored install") with a test; else document the second copy and word its alarm (R-1287) | CC |
## Storage & devices — 14 rows (P3 5, P4 9)
@@ -233,7 +234,7 @@ stopping line that lies.
| **R-1213** | Storage & devices | P4 | **Operator detail still on two household pages:** the registered drive card's `ext4 · /dev/sdb1 · Model` line (`templates/storage.html`; W14 hid only the agent drive cards), and the heading „Távoli mentés (3. mentés) — titkosított, offsite" with the English „offsite". Found by the night helper 2026-10-21 (W11/W14 follow-up). | OPEN | — | Move the line behind „Részletek"; „…titkosított, házon kívüli" | CC |
| **R-1275** | Storage & devices | P4 | **An English household sees the drive names in Hungarian** on the install page (and wherever a storage label shows): „Belső tárhely — 52.0 GB free (default)”, „Külső tárhely (hdd_1)” (catalog sweep 2026-10-24, `audits/catalog-sweep-2026-10-24/navidrome/page-install-en.txt`). The label is stored once, in Hungarian, when the drive is found (`felhom-controller/controller/internal/settings/settings.go``InferStorageLabel`); a household may rename it. | OPEN | — | Render a label that still equals the inferred default in the reader's language; a test per language | CC |
| **R-1276** | Storage & devices | P4 | **„Remove with data” leaves the app's empty parent folder** (`<drive>/appdata/<app>`) when the template binds a subfolder: Grimmory, RomM, Paperless-ngx (catalog sweep 2026-10-24; 4 KB each; the bound folders were removed). Harmless (empty, not counted as old data), untidy for a household that browses the drive. | OPEN | — | Remove the parent when it is empty after the bound folders go; a test | CC |
| **R-1282** | Storage & devices | P3 | **The „Később" 03:30 media restart is forgotten by a controller restart — store the due time (D44 #24).** The restart is an in-memory timer (`felhom-controller/controller/internal/web/media_events.go``handleMediaRestart`, `time.AfterFunc`); after a controller restart only the notice stays. Risk: media apps do not see the new drives until the household presses „now". Pick: store the due time and run the restart after a controller restart (a due time already past runs at once, within the same rules). Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §11. | OPEN | — | a small controller change + a test that a stored due time survives a restart and fires once; rides a controller release | CC |
| **R-1282** | Storage & devices | P3 | **The „Később" 03:30 media restart is forgotten by a controller restart — store the due time (D44 #24).** The restart is an in-memory timer (`felhom-controller/controller/internal/web/media_events.go``handleMediaRestart`, `time.AfterFunc`); after a controller restart only the notice stays. Risk: media apps do not see the new drives until the household presses „now". Pick: store the due time and run the restart after a controller restart (a due time already past runs at once, within the same rules). Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §11. |**VERIFY** — fixed on main 2026-10-25 night (controller `b5863a9`), waits for the controller release: the „later" time stored, resumed at start, once (`r1282_media_later_test.go`, red-proved). **Earlier state:** OPEN | — | a small controller change + a test that a stored due time survives a restart and fires once; rides a controller release | CC |
| **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f <conf>` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC |
| **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC |
| **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator |
| **R-1280** | Security & access | P3 | **A signed operation's window has no maximum length — cap it at 24 h (D44 #22).** The agent (`felhom-agent/internal/authz/verifier.go`) and the root wrapper (`configs/felhom-os-apply`) check `now ∈ [issued_at, expires_at]` with a 2-minute skew on `issued_at`, but neither checks `expires_at − issued_at`; `felhom-opsign` defaults to 30 min (`cmd/felhom-opsign/main.go:57`) and the nonce store keeps a record until its expiry. Risk: a signed job made with a long window by mistake stays usable for that whole time. Pick: the agent and the root wrapper refuse a window longer than 24 h. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/04-control-plane-authorization.md` §2. | OPEN | — | one agent change in both checkers + a test that a 24 h + 1 s window is refused (red first) and a 24 h one is accepted; rides an agent release (agent releases stop for the operator's vouch) | CC |
| **R-1281** | Security & access | P2 | **Every app container can reach FileBrowser directly; the only guard is the secret header name — take FileBrowser off the app networks (D44 #23).** FileBrowser trusts its proxy header (`auth.methods.proxy`); every app container can reach `filebrowser:80` without traefik; the guard is that the header NAME is a per-box secret that traefik's forwardAuth strips from client requests (`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`). Risk: an app that learns the name (a hacked app on the box) reads and writes every household file. Pick: FileBrowser joins only the network traefik uses, so no app container reaches it directly. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/13-threat-model.md` §5. | OPEN | — | a compose/network change in the controller's infra render + a test on the rendered networks; a live check on a Tier-0 box that a forged header from an app container no longer reaches FileBrowser (the 2026-10-18 measurement, now refused) while the dashboard's file manager still signs in; one controller release | CC |
| **R-1280** | Security & access | P3 | **A signed operation's window has no maximum length — cap it at 24 h (D44 #22).** The agent (`felhom-agent/internal/authz/verifier.go`) and the root wrapper (`configs/felhom-os-apply`) check `now ∈ [issued_at, expires_at]` with a 2-minute skew on `issued_at`, but neither checks `expires_at − issued_at`; `felhom-opsign` defaults to 30 min (`cmd/felhom-opsign/main.go:57`) and the nonce store keeps a record until its expiry. Risk: a signed job made with a long window by mistake stays usable for that whole time. Pick: the agent and the root wrapper refuse a window longer than 24 h. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/04-control-plane-authorization.md` §2. |**VERIFY** — fixed on main 2026-10-25 night (agent `d912307`), waits for the agent release + the operator's vouch: 24 h in the verifier, the root wrapper and felhom-opsign (red-proved, `redproof-r1280.txt`). **Earlier state:** OPEN | — | one agent change in both checkers + a test that a 24 h + 1 s window is refused (red first) and a 24 h one is accepted; rides an agent release (agent releases stop for the operator's vouch) | CC |
| **R-1281** | Security & access | P2 | **Every app container can reach FileBrowser directly; the only guard is the secret header name — take FileBrowser off the app networks (D44 #23).** FileBrowser trusts its proxy header (`auth.methods.proxy`); every app container can reach `filebrowser:80` without traefik; the guard is that the header NAME is a per-box secret that traefik's forwardAuth strips from client requests (`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`). Risk: an app that learns the name (a hacked app on the box) reads and writes every household file. Pick: FileBrowser joins only the network traefik uses, so no app container reaches it directly. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/13-threat-model.md` §5. |**VERIFY** — fixed on main 2026-10-25 night (controller `7ea942a`), waits for the controller release: FileBrowser served on `felhom-files` (traefik + OnlyOffice while linked), off traefik-public. **Live on the rig (rc.3):** before, both app containers signed in with the forged header (200); after, no answer (`000`); the household's „Fájlok ↗" walk 200, no cookie 401 (`rig/r1281-*.txt`). The migration live too: FileBrowser put back on traefik-public by hand → the next 5-minute round logged „left traefik-public — no app container reaches it any more" and the apps got no answer again. OnlyOffice's join by test only (not installed on the rig). On the release every box re-creates FileBrowser once (its compose comment changed). **Earlier state:** OPEN | — | a compose/network change in the controller's infra render + a test on the rendered networks; a live check on a Tier-0 box that a forged header from an app container no longer reaches FileBrowser (the 2026-10-18 measurement, now refused) while the dashboard's file manager still signs in; one controller release | CC |
| **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. |**VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | OPEN — **now in controller 0.343.0 (released 2026-10-25); the three edges act only on a box whose agent is ≥ 0.165.0 (it alone reports `waiting_for_quiet`) — fix before any household gets agent 0.165.0 by signed update** | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC |
| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). |**VERIFY** — fixed on main 2026-10-25 night (controller `a79a292`), waits for the controller release (before any household gets agent 0.165.0 by signed update): every running seeder held, a removed app's record dropped, qBittorrent's slot named (`up_limit`/`alt_up_limit`). Red-proofs `audits/night-2026-10-25/redproof-r1261.txt`. **Live on the night's rig** (BYO VM, agent 0.165.0, rc.1, two seeders): both held, 94.8 Mbit/s, both limits back; a speed-mode switch mid-test left no 1 KiB/s (`rig/hold-watch*.txt`); the removed-app case by test only. **Earlier state:** OPEN — **now in controller 0.343.0 (released 2026-10-25); the three edges act only on a box whose agent is ≥ 0.165.0 (it alone reports `waiting_for_quiet`) — fix before any household gets agent 0.165.0 by signed update** | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC |
| **R-1262** | Monitoring & notifications | P3 | **The upload watch treats ANY viewer as an explanation for any amount of upload, so a hacked Plex can send freely while one family member watches.** Found 2026-10-24 (SEC-1 X1): `felhom-controller/controller/internal/upwatch/watch.go:275,299`; `08` §9 states the rule, its limits list did not name this case. | OPEN | — | propose a cap (viewers × a bitrate ceiling) for the operator's R-1244 design; `08`'s limits list names the case meanwhile | CC |
| **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC****2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent``acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
@@ -312,6 +313,7 @@ stopping line that lies.
| **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
| **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
| **R-1287** | Monitoring & notifications | P3 | **A D47 set-aside mails the operator as „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" — no backup failed; the body says the right thing („its backup belonged to an earlier install … SET ASIDE whole").** Seen 2026-10-25 on Tester 1 with controller 0.343.0 (attended release run): two such mails at 17:31:56Z and 17:36:55Z. The controller sends the set-aside as `backup_run_failures` leg `unit-set-aside`, `Failed: 1` (so the hub never drops a line that claims zero); the hub's subject for that event is fixed („N of M apps failed to back up", `hub/internal/notify/templates.go:353`). The CLAUDE.md corollary: an alarm's text must say what it counts. After the floor, every household reinstall mails the operator a red „failed to back up". | **VERIFY** — fixed on main 2026-10-25 night, waits for the hub release (v0.171.0): a set-aside-only digest is a warning with „… set aside — nothing failed (<apps>)"; a run with a real failure stays red (`hub/internal/notify/r1287_setaside_test.go`, red-proved) | — | two-repo: the hub words a `unit-set-aside` leg on its own („… a backup of an earlier install was set aside — nothing failed", warning, not error) with a test on the rendered subject; or a new event type through the allow-list (`hub-event-allowlist-gotcha`); one hub release | CC |
| **R-1291** | Monitoring & notifications | P3 | **A held app gives the operator no lasting signal: one event at the stop, then nothing — demo-hp's Kimai sat held 11 days unseen.** Found 2026-10-25 night (R-1286). No held-app state reaches the hub (the report carries none; the hub shows none); the crash-loop stop sends `app_stopped_unhealthy` once. On a household box an app held after a crash loop, a failed update or a failed restore stays stopped until someone opens its page. | OPEN | — | a two-repo design: the controller reports its holds (stack, reason, since), the hub shows them on the host and mails a reminder after N days (the D-level of the reason); one row's worth of design first | CC |
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-10-26 (morning)
**Updated 2026-10-25 (evening, attended release).** Every box runs the new controller: a restore that does not start says so and keeps both copies (D49), a reinstall never touches the old backup, a restore downloads first. The hub stores the recovery package in one step; the new agent is vouched; the catalog has three fixes.
## Moved from STATUS on 2026-10-25 (attended release run)
**Updated 2026-10-25 (Sunday).** A reinstall keeps old backups whole; a restore downloads first and tells the truth.
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
admin committed