Night 2026-10-25 Part 4–5: each fix's [DESIGN] line (04, 05, 07, 08, 11, 13, 15), the register (R-1261, R-1281, R-1282, R-1288, R-1280, R-1285, R-1283, R-1287, R-1289, R-1278 to VERIFY; R-1286 waiting on the operator; R-1291 and R-1292 opened), STATUS (D51 Kimai, the Monday list), the night report with its three-layer teardown, and the rig evidence (scanned: 0 known-secret hits)
gates / gates (push) Successful in 13m42s

This commit is contained in:
admin committed 2026-10-25 23:47:13 +01:00
1 parent f63284861e
commit 9258cdd11c
33 files changed
+57024 -22

No files matched your search

+14
View File
@@ -16,6 +16,20 @@
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
> **2026-10-25 night (unattended, commit only) — the night's fixes, each red-proved; proven live on a BYO test VM (demo-hp
> VM 9372, customer Tester4, agent 0.165.0, controller rc.1–rc.3).** Controller (`a79a292`…`7ea942a`, rides v0.344.0):
> R-1261 the line hold holds EVERY running seeder, drops a removed app's record, names qBittorrent's slot (`up_limit` /
> `alt_up_limit` via `app/preferences` — `transfer/uploadLimit` follows the current speed mode, measured); R-1288 the
> restore of a set-aside unit sets the replaced unit aside AT the restore, no alarm (the brief's safest reading, `07`
> §8.2d changed); R-1282 the „later" media restart stored and resumed; R-1281 FileBrowser served on its own
> `felhom-files` network (traefik + OnlyOffice while linked), off traefik-public — the forged header from an app went
> 200 → no answer; R-1286 every hold reason says what it is (Kimai's was a crash-loop stop, not a failed restore). Hub
> (`f6328486`, v0.171.0): R-1287 a set-aside-only digest is a warning „nothing failed"; R-1289 suffixed versions never
> „the fleet's newest". Agent (`d912307`, `abcc1e6`, next agent release + vouch): R-1280 a signed window ≤ 24 h; R-1285
> the agent runs the wrapper's `bundle-policy` pass after every bundle install. Drill catalog `6733764`: R-1283 MeTube
> `optional` (a named ruling in the download-binds gate). New rows R-1291 (a held app has no lasting signal), R-1292
> (the 50 % rule follows the speed mode).
> **2026-10-25 (evening, attended) — D50 = A; installer 1.36.1 published.** (`09` §3 decision 262.) The registry keeps the
> newest 5 agent versions + the MinAgent floor + the day-0 agent; the agent CI asserts exactly these
> (`retention-policy.json` `pinned`, copied from hub state by the vouching session — CI holds no hub secret). The operator
+49
View File
@@ -0,0 +1,49 @@
# REPORT — installer 1.36.1 published; the night's fixes (2026-10-25 evening → 2026-10-26 morning)
**Plain words first.**
1. **Every package that is still named is there.** You pruned old controller, agent and golden packages by hand; nothing a
box or the installer can still ask for is missing (the agent 0.165.0 + bundle, the 0.158.1 floor agent, golden
0.331.0, the agent tags from 0.158.1 up, controller 0.343.0 / 0.342.0 / 0.331.0, samba 1.2.0, the ISO).
2. **Installer 1.36.1 is live, and its three messages were seen:** a wrong customer → „not found (404)", a wrong
passphrase → „passphrase REJECTED (401)", a customer whose box is live → „already has a live box (409)" (R-1167's
sentence, which could never show before).
3. **The agent CI is green with D50 = A** (the newest 5 versions + the MinAgent floor + the day-0 agent).
4. **R-1261 is fixed and proven before Tuesday** — on a test VM with two seeders: both held, 94.8 Mbit/s measured, both
limits back; a speed-mode switch mid-test never left 1 KiB/s. It reaches households with Monday's controller release.
5. **You decide:** Kimai on demo-hp — press Start, or remove it (STATUS, D51). And the Monday release list (STATUS).
## Each fix, before → after
- **R-1261 (controller `a79a292`):** one seeder held, a removed app's record kept forever, qBittorrent's limit put back
in whatever slot its mode showed → every seeder held, the record dropped, the held slot named. Live on the rig.
- **R-1287 (hub `f6328486`, v0.171.0):** a set-aside mailed „🔴 1 of 1 apps failed to back up" → „⚠️ … a backup of an
earlier install was set aside — nothing failed", a warning. Test on the real dispatch path.
- **R-1288 (controller `b5863a9`):** a set-aside restore's replaced unit was set aside at the NEXT backup, with a second
alarm → set aside at the restore, no alarm. Live on the rig (one set-aside mail only).
- **R-1281 (controller `7ea942a`):** an app container signed in to FileBrowser with a forged header (200) → no answer;
the household's file manager still 200; a FileBrowser left on traefik-public is moved off by the next round. Live.
- **R-1289 (hub `f6328486`):** a dev build no box ran was „the fleet's newest" → test builds never count.
- **R-1282 (controller `b5863a9`):** the 03:30 media restart was forgotten by a controller restart → stored, resumed once.
- **R-1280 (agent `d912307`):** a signed window of any length → at most 24 h (verifier, root wrapper, felhom-opsign).
- **R-1285 (agent `abcc1e6`):** a BYO host lost our SSH file one update late → the agent runs the new wrapper's policy
pass after every bundle install.
- **R-1283 (drill catalog `6733764`):** MeTube's downloads `excluded` → `optional` (a named ruling in the gate).
- **R-1286 (controller `838c834`):** Kimai's crash-loop hold read „a failed restore whose rollback also failed" → every
hold reason says what it is. The cause: the crash-loop guard during the 10-14 power drill.
## Teardown — three layers
- **Machine (rig VM 9372 on demo-hp):** the product's own `--uninstall` ran first (rc 0, „UNINSTALL complete — removed:
guest 9201, the felhom-agent …", `audits/night-2026-10-25/rig/uninstall-console.txt`); the host journal was copied off
before (`rig/rig-host-journal.txt`). Then `qm destroy 9372 --purge 1`: `qm list` shows no 9372, its folder on
`nvme-scratch` is gone, the plain-PVE ISO is removed from demo-hp, the ISO work folder on DooPlex is removed, the old
host key is removed from `known_hosts`.
- **Host (demo-hp):** guest 9202's test tracker, downloader container, test files and image are removed (0 containers,
0 files, port 6969 closed). `nvme-scratch` is a standing storage (it holds 341, 9202, 9401) — this session did not add
it, so it stays. `local-lvm` (the thin pool) was never used: 48 %.
- **Hub:** host `Tester4-84bbe7` stays on the Hosts page — a host delete is the operator's (STATUS, „Also Monday"). The
R-1285 test package was deleted from the registry (204, then 404).
- **Secrets:** the scratch session, CSRF and header files, the dashboard password file, the rig root password file, the
test bundle and the test agent build are shredded. The evidence was scanned against the four known values (through
`sec`) and four token shapes: 0 hits (one shape hit is code that reads a file). The `sec` entries
`RIG_2026_10_25_DASH_PW` and `RIG_2026_10_25_SETUP_CODE` remain; `sec` has no delete, and the box they opened is gone.
+4 -2
View File
@@ -2,7 +2,7 @@
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-10-25 (evening, attended release).** Every box runs the new controller: a restore that does not start says so and keeps both copies (D49), a reinstall never touches the old backup, a restore downloads first. The hub stores the recovery package in one step; the new agent is vouched; the catalog has three fixes.
**Updated 2026-10-26 (morning).** Installer 1.36.1 is live: a wrong passphrase now says so. Overnight, commit only: the torrent line test holds every seeder; FileBrowser is off the app network; a set-aside mails as a set-aside; the four D44 picks are built and proven on a test VM.
## What works
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
@@ -31,6 +31,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|---|---|---|---|---|---|---|
| D51 | R-1286 | demo-hp's Kimai was stopped by the crash-loop guard on 10-14 (the power drill) and is still held. Start it again, or remove it? | **Press Start** (it lifts this kind of hold; if it crash-loops again it is held again and we look) | remove it through the product: the test app and its data are gone | it stays stopped; nothing else waits on it | — |
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
@@ -47,7 +48,8 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
**Day, 2026-10-23 — your do list.**
**Monday 2026-10-26 (attended; each a STOP).** 1. Hub v0.171.0 (R-1287, R-1289). 2. Agent v0.166.0 + your vouch (R-1280, R-1285). 3. Controller v0.344.0 — before any household gets agent 0.165.0 (R-1261, R-1281, R-1288, R-1282, R-1286, R-1278). 4. Catalog: MeTube (R-1283). 5. Dated: R-1183 today, R-1250 Tuesday. 6. Hub Hosts page: delete the dead `Tester4-84bbe7`. If not: the fixes stay unreleased; the host alarms.
**2026-10-23 do list.**
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
2. On the hub's Hosts page: Discard appliance 49. If not: a dead entry alarms.
3. Give the homelab session R-1241, R-1242, the lists at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` and `documentation/audits/dooplex-alarms-2026-10-24/README.md`, and an upload alarm on DooPlex's `eno1` (~100 Mbit/s line; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups.
@@ -80,7 +80,9 @@ nonce (DoS-safe). Each layer is mandatory and was proven to reject independently
the root wrapper (`configs/felhom-os-apply`) alike (corrected 2026-10-24). **[DESIGN — the operator picked a code change
2026-10-25 (D44); row R-1280]:** neither caps the window's LENGTH (`expires_at − issued_at`); `felhom-opsign` defaults to 30 min
(`cmd/felhom-opsign/main.go:57`), and the nonce store keeps a record until its expiry. The pick: the agent and the
root wrapper refuse a window longer than 24 h. Not built yet;
root wrapper refuse a window longer than 24 h. **Built 2026-10-25 night (agent `d912307`, unreleased — rides the next
agent release):** `authz.MaxWindow = 24h` (`ErrWindowTooLong`), the wrapper's `MAX_WINDOW_S`, and `felhom-opsign -ttl`
refuses more; exactly 24 h is accepted;
- **nonce** — unseen within the window (the nonce store **must be persistent across agent restarts**
and expiry-pruned; a non-persistent store reopens the replay window after a restart).
@@ -139,7 +139,9 @@ walk is green; the golden is baked weekly and vouched by the operator; a new box
itself to the floor (R-1052: not yet proven end to end). Setting the global floor today needs the operator page with the
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **„The fleet" is the
newest RELEASED controller any box reported** — a test build (`-rc.N`, `-dev…`) never counts (R-1289, hub v0.171.0,
unreleased: the semver compare could not read a suffix and kept a dev build no box ran as the fleet's newest). **The agent list
offers real agent packages only (R-1011, hub 0.162.0, deployed since; corrected 2026-10-24):** the sha lookup asks for exactly the
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
@@ -1720,7 +1720,7 @@ names the app and both identities). Nothing moves a unit while that app is being
telepítés mentése") with **„Visszaállítás"** — the one unit-restore body, with the unit's own database password — and
„Eltávolítás". Only those two, and the app's remove with „delete its backups too", ever remove it (the §8.2a / R-1252 H9
rule). After a restore from it, the app is that unit's install; the app's own unit of the install it replaced is then
set aside at the next backup — never lost.
set aside ~~at the next backup~~ **at the restore itself** — never lost. **[DESIGN — R-1288, 2026-10-25 night, the brief's safest reading; controller `b5863a9`, unreleased]:** the restore of a set-aside unit moves the replaced install's unit aside whole right after it (success, or „not started yet" — the data stays, D49 = A), one log line „replaced by the restored install", **no operator alarm** — the household chose this restore; before, the next backup's writer set it aside with a second alarm and a second kept copy (measured on Tester 1, 0.343.0). **[FACT]** on the night's rig (rc.2): the line, the account back, the next backup set nothing aside, one set-aside mail only (`audits/night-2026-10-25/rig/r1288-live.txt`). A set-aside's operator mail (R-1287, hub v0.171.0, unreleased) reads „⚠️ … a backup of an earlier install was set aside — nothing failed", a warning.
**Known limit, stated.** A box that reinstalled and refreshed BEFORE this release already holds a rewritten unit; its
old password is gone and nothing can bring it back. A unit from before D47 of an app with no carried secret has no
@@ -1728,7 +1728,7 @@ signal: it is treated as this install's (the old behaviour) and gets the identit
**Why a volume app's reinstall asks no kept-data question:** the question (§6.5, decision 36) is about the app's drive
FOLDER (`${HDD_PATH}` binds under `appdata/`); a volume app keeps nothing there — „remove, keep my data" removes its named
volumes and keeps only its backups — so the install starts empty, and the kept backup is now the set-aside unit on the
restore page. (The missing offer at install is its own row, R-1278.)
restore page. **[DESIGN — R-1278, controller `fbec5cb`, unreleased]:** the install page now says so for an app not installed — „the new install starts empty; load the backup afterwards on the Restore page" — with a link (set-aside and removed-app units counted).
### 8.2e A restore whose app does not start is never „success" [DESIGN, R-1270, 2026-10-25 — controller `b680db8` + fix A; D49 = A (operator 2026-10-25); released in controller 0.343.0]
@@ -591,6 +591,15 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me
start-up check puts it back; the 50 % rule waits while a record exists. Measured on demo-hp with a real seed (10
Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; with it the uplink fell to 0.02–0.16 Mbit/s,
94.2 Mbit/s measured, the app held 56.8 s and back to 1 249 280 B/s exactly (`audits/dooplex-alarms-2026-10-24/`).
**[DESIGN — R-1261, 2026-10-25 night; controller `a79a292`, unreleased]:** EVERY running seeder is held (qBittorrent
and Transmission — one left seeding kept the test refused `not-quiet` every time); all records first
(`upwatch.line_test_holds`), then the apps; a record whose app was removed while held is dropped (it used to stop the
50 % rule for good); qBittorrent's hold names its slot (`up_limit` / `alt_up_limit` through `app/preferences` —
`transfer/uploadLimit` follows the CURRENT speed mode, measured on 5.2.4), so the household's switch to the
alternative speed during the test never leaves 1 KiB/s behind. **[FACT]** the night's rig (BYO VM, agent 0.165.0,
rc.1, two seeders to 9202's qBittorrent): with a mode switch mid-test both limits came back, the test refused
(the switch let qBittorrent upload); without, both held (qBittorrent in its alternative slot), 94.8 Mbit/s measured,
both limits back exactly (`audits/night-2026-10-25/rig/hold-watch*.txt`).
While the box owns it, qBittorrent's page says: „A qBittorrent
feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
6. **Limits, stated:** *(added 2026-10-24, review round 2: ANY viewer explains any amount of upload, so a hacked app can send freely while one real playback runs — `upwatch/watch.go:275,299`; R-1262)*; an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
+5 -1
View File
@@ -79,7 +79,11 @@ boxes installed. Nobody keeps a hand-written list.
**[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased]** The host's root-key-only sshd
drop-in (D4, R-982) is an APPLIANCE-only file of the config bundle: written only when the root-owned install record
says `appliance`; on a `byo` record nothing is written and a copy the bundle wrote earlier (this path, our first line)
is removed; a box whose record proves nothing is left as it is (R-1257).
is removed; a box whose record proves nothing is left as it is (R-1257). **[FACT] 2026-10-25 (BYO test host, agent
0.164.0 → 0.165.0):** the removal came one bundle LATE — the bundle that brings the new wrapper is planned by the OLD
one (R-1285). **[DESIGN — R-1285, agent `abcc1e6`, unreleased]:** right after every bundle install the agent asks the
wrapper now on disk for a policy pass (`mode: "bundle-policy"`), which removes our appliance-only files from a BYO
record (no signature: it only removes a file we wrote; a failing `sshd -t` puts it back).
- **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5).
- **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in
the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since
@@ -84,7 +84,13 @@ directly; the only guard is that the header NAME is a per-box secret, which trae
client request and sets only for a browser holding the dashboard's files pass
(`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`, `internal/web/files_sso.go`). The name file's
creation race was fixed by review round 2 (controller `b01bb4b`). The pick: FileBrowser joins only the network traefik uses, so no app
container reaches it directly. Not built yet.
container reaches it directly. **Built 2026-10-25 night (controller `7ea942a`, unreleased):** a network of its own for
the route, `felhom-files` — traefik is on it, the CHECKED FileBrowser joins it by id after R-1126's default-login check,
and FileBrowser leaves traefik-public; OnlyOffice joins it while — and only while — its FileBrowser link is on (it
fetches and saves documents at `http://filebrowser:80`), the one stated exception. **[FACT] measured on the night's rig:**
before, from the qBittorrent and Transmission containers the forged header as the household user got the drive's
listing (200); after, no answer (`000`); the household's „Fájlok ↗" walk still gets 200 through traefik
(`audits/night-2026-10-25/rig/r1281-*.txt`).
## 6. What we accept (do not re-report)
@@ -169,8 +169,10 @@ download (R-1019). *(Fixed since: catalog `c50adfc` marks the download and watch
`app-catalog-felhom.eu/templates/transmission/.felhom.yml:46-51`; noted 2026-10-24.)* **[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1283]:** the same commit also
marks the download folders of NZBGet, SABnzbd and MeTube `excluded` (`templates/{nzbget,sabnzbd,metube}/.felhom.yml`).
NZBGet and SABnzbd stay excluded (their files can be fetched again). The pick for MeTube: `optional`, so the
second-drive copy carries it and the off-site copy does not — a MeTube video can be gone from its source site. Not
built yet. **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
second-drive copy carries it and the off-site copy does not — a MeTube video can be gone from its source site.
**Built in the DRILL catalog 2026-10-25 night (`app-catalog-drill` `6733764`, not live):** `class: optional`, and the
download-binds gate allows `optional` only for an app named with its ruling (`OPTIONAL_BY_RULING`); mandatory and
unlisted stay refused. The live push is the operator's (the Monday list). **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
media is no single app's data; whether the library gets its own household-level backup is a separate decision.
## 8. Removal and export assume one drive per app
@@ -275,7 +277,9 @@ returns** (G7f).
**[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1282]:** the „Később"
03:30 media restart is an IN-MEMORY timer — a controller restart forgets it and only the notice stays
(`internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`). The pick: the controller stores the due
time and runs the restart after a restart of its own. Not built yet.
time and runs the restart after a restart of its own. **Built 2026-10-25 night (controller `b5863a9`, unreleased):**
`settings.media_restart_later_at`; at start `ResumeMediaLater` re-arms it, runs it at once when it passed while the
controller was down, and clears it once it ran.
**[DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]:** the Sonarr/Radarr API key is READ from
`/config/config.xml`; the „set at install" option §6 calls cleaner was not built (`internal/mediawire/mediawire.go:336-345`).
@@ -0,0 +1,4 @@
# R-1278 red-proof: the notice's {{if}} made false, 2026-10-25 night
=== RUN TestR1278_TheInstallPageOffersTheKeptBackup
r1278_kept_backup_offer_test.go:17: with a kept backup of an earlier install the page must say so and link the restore page
--- FAIL: TestR1278_TheInstallPageOffersTheKeptBackup (0.06s)
@@ -0,0 +1,9 @@
# R-1280 red-proofs: the 24 h checks switched off, 2026-10-25
verify_test.go:128: want authz: op validity window too long, got <nil>
--- FAIL: TestVerify_RejectsPerStage (0.00s)
--- FAIL: TestVerify_RejectsPerStage/window_longer_than_24_h (0.00s)
FAIL
FAIL: test_a_window_longer_than_24h_is_refused (test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused)
AssertionError: 0 != 2 : {'authority': 'signed', 'docker_engine': '29.7.2', 'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'running'}, 'felhom-controller': {'health': 'healthy', 'id': 'aaa111', 'state': 'running'}}, 'controller': 'healthy', 'controller_docker_ok': True, 'docker_ok': True, 'network_ok': True}, 'health_before': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'running'}, 'felhom-controller': {'health': 'healthy', 'id': 'aaa111', 'state': 'running'}}, 'controller': 'healthy', 'controller_docker_ok': True, 'docker_ok': True, 'network_ok': True}, 'installed': [{'name': 'bash', 'origin': 'Debian', 'version': '5.2.37-2+b9'}, {'name': 'containerd.io', 'origin': 'Debian', 'version': '2.3.6-1~debian.13~trixie'}, {'name': 'docker-ce', 'origin': 'Debian', 'version': '5:29.8.2-1~debian.13~trixie'}, {'name': 'libc6', 'origin': 'Debian', 'version': '2.41-12+deb13u3'}, {'name': 'openssl', 'origin': 'Debian', 'version': '3.5.6-1~deb13u1'}], 'layer': 'docker', 'mode': 'apply', 'oom_check': {'detail': 'the engine reported the memory kill: OOMKilled=true and the oom event', 'exit_code': 137, 'image': 'gitea.dooplex.hu/admin/felhom-controller:0.300.0', 'oom_event': True, 'oom_killed': True, 'result': 'pass'}, 'pass_seconds': 0.0, 'pending': [{'from': '5.2.37-2+b9', 'name': 'bash', 'origin': ['Debian'], 'to': '5.2.37-2+b10'}], 'plan': {'already': 0, 'from_snapshot': 0, 'not_installed': 0, 'upgrade': 2}, 'reboot_needed': False, 'reboot_scanned': True, 'refused': None, 'release_id': 'os-docker-t1', 'repair': {'clean_after': True, 'fixed': 0, 'half_configured_before': 0, 'journal_before': 0}, 'restart_needed': [], 'seconds': 0.0, 'socket_restarted': ['felhom-controller'], 'undo': False, 'upgraded': [{'name': 'docker-ce', 'version': '5:29.8.2-1~debian.13~trixie'}, {'name': 'containerd.io', 'version': '2.3.6-1~debian.13~trixie'}], 'vmid': 9201}
Ran 1 test in 0.013s
FAILED (failures=1)
@@ -0,0 +1,9 @@
# R-1281 red-proofs, 2026-10-25 night
## 1. the traefik-public disconnect skipped
=== RUN TestR1281_FileBrowserLeavesTheAppNetworkAndIsServedOnTheFilesNetwork
filebrowser_password_test.go:341: FileBrowser is still on traefik-public — every app container reaches it
--- FAIL: TestR1281_FileBrowserLeavesTheAppNetworkAndIsServedOnTheFilesNetwork (0.00s)
## 2. the OnlyOffice step removed
=== RUN TestR1281_OnlyOfficeJoinsTheFilesNetworkOnlyWhileLinked
filebrowser_password_test.go:385: a linked OnlyOffice must reach FileBrowser: map[3f2a9c@felhom-files:true felhom-controller@felhom-filebrowser:true traefik@felhom-files:true]
--- FAIL: TestR1281_OnlyOfficeJoinsTheFilesNetworkOnlyWhileLinked (0.00s)
@@ -0,0 +1,7 @@
# R-1283 red-proof: the ruling's entry removed (drill catalog), 2026-10-25 night
download-binds gate FAILED (1 of 6 download folder(s)):
templates/metube: the download folder media/metube is declared `optional` — a nightly copy on the second drive; declare it `class: excluded` (R-1019)
XX GENUINE (R-1283): metube's folder optional by ruling rc=1 (expected 0)
ok FACT (R-1283): metube's folder mandatory — the ruling allows optional only rc=1 (expected 1)
ok FACT (R-1283): metube's folder UNLISTED rc=1 (expected 1)
FAIL: GENUINE (R-1283): metube's folder optional by ruling: rc=1 expected 0; missing ['every one']
@@ -0,0 +1,10 @@
# R-1285 red-proofs, 2026-10-25 night
## agent: the policyPass call removed
=== RUN TestR1285_AnInstalledBundleIsFollowedByThePolicyPass
bundle_test.go:173: after the install the agent must run the policy pass (host layer, no envelope): [map[bundle:/tmp/TestR1285_AnInstalledBundleIsFollowedByThePolicyPass3671177433/001/bundle-0.143.0.json layer:host mode:bundle release_id:bundle-0.143.0 signed:map[blob_b64:eyJvcCI6ImFnZW50X2NvbmZpZ191cGRhdGUifQ== sig:SIG]]]
--- FAIL: TestR1285_AnInstalledBundleIsFollowedByThePolicyPass (0.00s)
## wrapper: policy_pass made to do nothing
FAIL: test_the_policy_pass_removes_our_copy_from_a_byo_host (test_felhom_config_bundle.ByoKeepsItsSSH.test_the_policy_pass_removes_our_copy_from_a_byo_host)
AssertionError: '/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf' unexpectedly found in {'/var/lib/felhom-install/state.json': b'{"completed": [], "mode": "byo"}', '/etc/felhom/os-trust.json': b'{
Ran 1 test in 0.057s
FAILED (failures=1)
@@ -0,0 +1,4 @@
# R-1286 red-proof: holdWhy back to the old two-way branch, 2026-10-25 night
=== RUN TestR1286_EveryHoldReasonSaysWhatItIs
r1286_hold_why_test.go:32: kimai (reason "unhealthy_stop"): the gate says "held after a failed restore whose rollback also failed (2026-10-14T09:06:31Z) — clear the hold to start it", want it to contain "stopped by the box for a crash loop"
--- FAIL: TestR1286_EveryHoldReasonSaysWhatItIs (0.00s)
@@ -0,0 +1,12 @@
# R-1288 red-proof (retireReplacedUnit call removed), 2026-10-25
=== RUN TestR1288_ARestoreOfASetAsideUnitRetiresTheReplacedUnitQuietly
unit_owner_test.go:358: after the restore the primary place still holds install "bbbb000000000002"'s unit — it must be set aside at the restore
--- FAIL: TestR1288_ARestoreOfASetAsideUnitRetiresTheReplacedUnitQuietly (0.02s)
# R-1282 red-proof (ResumeMediaLater a no-op)
=== RUN TestR1282_ALaterRestartThatPassedWhileDownRunsOnceAtStart
r1282_media_later_test.go:43: the passed restart ran 0 time(s) at start, want 1
--- FAIL: TestR1282_ALaterRestartThatPassedWhileDownRunsOnceAtStart (2.00s)
=== RUN TestR1282_AFutureLaterRestartIsKeptOverAStart
r1282_media_later_test.go:74: the timer must be armed for 2026-10-25 22:29:25.783026813 +0100 CET m=+3602.004971479, got 0001-01-01 00:00:00 +0000 UTC (armed false)
--- FAIL: TestR1282_AFutureLaterRestartIsKeptOverAStart (0.05s)
@@ -0,0 +1,33 @@
20:28:10 hold seen — NO switch this time
qB prefs: up_limit 2499584 alt_up_limit 1024 mode=1
qBittorrent: [('cc-seed-qb.bin', 'uploading', 529, 'KiB/s')] limit=1024 mode=1
Transmission: [('cc-seed-tr.bin', 6, 1, 'KiB/s', 1.0)] {'speed-limit-up': 1, 'speed-limit-up-enabled': True}
qBittorrent: [('cc-seed-qb.bin', 'uploading', 339, 'KiB/s')] limit=1024 mode=1
Transmission: [('cc-seed-tr.bin', 6, 1, 'KiB/s', 1.0)] {'speed-limit-up': 1, 'speed-limit-up-enabled': True}
== controller lines
2026/10/25 20:28:03 [INFO] [upwatch] line test: qbittorrent held at {Bytes:1024 Alt:true Enabled:false KBps:0} (was {Bytes:3072000 Alt:true Enabled:false KBps:0})
2026/10/25 20:28:03 [INFO] [upwatch] line test: transmission held at {Bytes:0 Alt:false Enabled:true KBps:1} (was {Bytes:0 Alt:false Enabled:false KBps:500})
2026/10/25 20:28:03 [INFO] [upwatch] line test: 2 app(s) held — asking the agent for a test
2026/10/25 20:29:00 [INFO] [upwatch] qbittorrent's upload limit is back as it was ({Bytes:3072000 Alt:true Enabled:false KBps:0})
2026/10/25 20:29:00 [INFO] [upwatch] transmission's upload limit is back as it was ({Bytes:0 Alt:false Enabled:false KBps:500})
2026/10/25 20:29:00 [INFO] [upwatch] line test on request: 94.8 Mbit/s
2026/10/25 20:29:00 [INFO] [upwatch] the host's weekly line test: 94.8 Mbit/s (measured 2026-10-25T20:28:18Z)
== agent lines
Oct 25 12:41:54 pf-test felhom-agent[1117]: time=2026-10-25T20:41:54.210+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backu
Oct 25 12:48:11 pf-test felhom-agent[11486]: time=2026-10-25T12:48:11.728-07:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a back
Oct 25 13:13:03 pf-test felhom-agent[11486]: time=2026-10-25T13:13:03.535-07:00 level=INFO msg="local-api: line test requested" vmid=9201
Oct 25 13:13:18 pf-test felhom-agent[11486]: time=2026-10-25T13:13:18.538-07:00 level=INFO msg="linetest: test on request refused — the uplink is not quiet" mbps=10.0
Oct 25 13:13:18 pf-test felhom-agent[11486]: time=2026-10-25T13:13:18.539-07:00 level=INFO msg="local-api: line test request refused" vmid=9201 reason=not-quiet detail="the uplink sent 10.0 Mbit/s while held"
Oct 25 13:28:03 pf-test felhom-agent[11486]: time=2026-10-25T13:28:03.644-07:00 level=INFO msg="local-api: line test requested" vmid=9201
Oct 25 13:28:11 pf-test felhom-agent[11486]: time=2026-10-25T13:28:11.729-07:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=line-test
Oct 25 13:28:18 pf-test felhom-agent[11486]: time=2026-10-25T13:28:18.647-07:00 level=INFO msg="linetest: test on request (a held torrent app, R-1250)" server=10.77.0.1:5201
Oct 25 13:28:29 pf-test felhom-agent[11486]: time=2026-10-25T13:28:29.021-07:00 level=INFO msg="linetest: run 1: 94.8 Mbit/s" server=10.77.0.1:5201 server_bytes=119144448 server_seconds=10.058812
Oct 25 13:28:44 pf-test felhom-agent[11486]: time=2026-10-25T13:28:44.399-07:00 level=INFO msg="linetest: run 2: 94.7 Mbit/s" server=10.77.0.1:5201 server_bytes=119013376 server_seconds=10.057222
Oct 25 13:28:59 pf-test felhom-agent[11486]: time=2026-10-25T13:28:59.786-07:00 level=INFO msg="linetest: run 3: 94.4 Mbit/s" server=10.77.0.1:5201 server_bytes=118751232 server_seconds=10.067068
Oct 25 13:28:59 pf-test felhom-agent[11486]: time=2026-10-25T13:28:59.786-07:00 level=INFO msg="linetest: upload line measured: 94.8 Mbit/s (best of 3)" server=10.77.0.1:5201 runs=3 duration_s=41.138682127
== after
qB prefs: up_limit 2499584 alt_up_limit 3072000 mode=1
qBittorrent: [('cc-seed-qb.bin', 'uploading', 541, 'KiB/s')] limit=3072000 mode=1
Transmission: [('cc-seed-tr.bin', 6, 360, 'KiB/s', 1.0)] {'speed-limit-up': 500, 'speed-limit-up-enabled': False}
qBittorrent: [('cc-seed-qb.bin', 'uploading', 565, 'KiB/s')] limit=3072000 mode=1
Transmission: [('cc-seed-tr.bin', 6, 352, 'KiB/s', 1.0)] {'speed-limit-up': 500, 'speed-limit-up-enabled': False}
@@ -0,0 +1,27 @@
20:13:07 hold seen — switching qBittorrent to its alternative speed
qBittorrent: [('cc-seed-qb.bin', 'uploading', 258, 'KiB/s')] limit=3072000 mode=1
Transmission: [('cc-seed-tr.bin', 6, 1, 'KiB/s', 1.0)] {'speed-limit-up': 1, 'speed-limit-up-enabled': True}
qB prefs: up_limit 1024 alt_up_limit 3072000 mode=1
qBittorrent: [('cc-seed-qb.bin', 'uploading', 210, 'KiB/s')] limit=3072000 mode=1
Transmission: [('cc-seed-tr.bin', 6, 1, 'KiB/s', 1.0)] {'speed-limit-up': 1, 'speed-limit-up-enabled': True}
== controller lines
2026/10/25 19:58:03 [INFO] [upwatch] qBittorrent's upload limit is the household's — the box leaves it alone from now on
2026/10/25 20:13:03 [INFO] [upwatch] line test: qbittorrent held at {Bytes:1024 Alt:false Enabled:false KBps:0} (was {Bytes:2499584 Alt:false Enabled:false KBps:0})
2026/10/25 20:13:03 [INFO] [upwatch] line test: transmission held at {Bytes:0 Alt:false Enabled:true KBps:1} (was {Bytes:0 Alt:false Enabled:false KBps:500})
2026/10/25 20:13:03 [INFO] [upwatch] line test: 2 app(s) held — asking the agent for a test
2026/10/25 20:13:18 [INFO] [upwatch] qbittorrent's upload limit is back as it was ({Bytes:2499584 Alt:false Enabled:false KBps:0})
2026/10/25 20:13:19 [INFO] [upwatch] transmission's upload limit is back as it was ({Bytes:0 Alt:false Enabled:false KBps:500})
2026/10/25 20:13:19 [INFO] [upwatch] line test on request refused by the agent: not-quiet (the uplink sent 10.0 Mbit/s while held)
== agent lines
Oct 25 12:25:22 pf-test felhom-agent[2092]: time=2026-10-25T20:25:22.507+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backu
Oct 25 12:41:54 pf-test felhom-agent[1117]: time=2026-10-25T20:41:54.210+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backu
Oct 25 12:48:11 pf-test felhom-agent[11486]: time=2026-10-25T12:48:11.728-07:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a back
Oct 25 13:13:03 pf-test felhom-agent[11486]: time=2026-10-25T13:13:03.535-07:00 level=INFO msg="local-api: line test requested" vmid=9201
Oct 25 13:13:18 pf-test felhom-agent[11486]: time=2026-10-25T13:13:18.538-07:00 level=INFO msg="linetest: test on request refused — the uplink is not quiet" mbps=10.0
Oct 25 13:13:18 pf-test felhom-agent[11486]: time=2026-10-25T13:13:18.539-07:00 level=INFO msg="local-api: line test request refused" vmid=9201 reason=not-quiet detail="the uplink sent 10.0 Mbit/s while held"
== after
qB prefs: up_limit 2499584 alt_up_limit 3072000 mode=1
qBittorrent: [('cc-seed-qb.bin', 'uploading', 658, 'KiB/s')] limit=3072000 mode=1
Transmission: [('cc-seed-tr.bin', 6, 208, 'KiB/s', 1.0)] {'speed-limit-up': 500, 'speed-limit-up-enabled': False}
qBittorrent: [('cc-seed-qb.bin', 'uploading', 690, 'KiB/s')] limit=3072000 mode=1
Transmission: [('cc-seed-tr.bin', 6, 32, 'KiB/s', 1.0)] {'speed-limit-up': 500, 'speed-limit-up-enabled': False}
@@ -0,0 +1,13 @@
header name read: yes
transmission (networks: traefik-public ): no header -> 000 | forged header (household user) -> 000
qbittorrent (networks: traefik-public ): no header -> 000 | forged header (household user) -> 000
filebrowser networks: felhom-filebrowser felhom-files
traefik networks: felhom-files felhom-tunnel traefik-public
HTTP/1.1 200 OK
1. dashboard /__files/open -> HTTP/2 302 | to files host: True
2. files host /_felhom/sso -> HTTP/2 302 | files cookie set: True
3. the drive's listing through traefik with the files cookie -> 200
4. the same without any cookie (control) -> 401
felhom-filebrowser felhom-files
transmission (networks: traefik-public ): no header -> 000 | forged header (household user) -> 000
qbittorrent (networks: traefik-public ): no header -> 000 | forged header (household user) -> 000
@@ -0,0 +1,7 @@
header name read: yes
transmission (networks: traefik-public ): no header -> 401 | forged header (household user) -> 200
qbittorrent (networks: traefik-public ): no header -> 401 | forged header (household user) -> 200
filebrowser networks: felhom-filebrowser traefik-public
traefik networks: felhom-tunnel traefik-public
2026/10/25 22:10:12 GET | 401 | 172.18.0.5 | N/A | 0ms | "/api/resources?path=%2F&source=Adat"
2026/10/25 22:10:12 GET | 200 | 172.18.0.5 | felhom-hou.. | 1ms | "/api/resources?path=%2F&source=Adat"
@@ -0,0 +1,4 @@
candidates {'data': {'vmid': 9201, 'initialize': [{'device': '/dev/sdc', 'size_bytes': 17179869184, 'model': 'QEMU HARDDISK', 'data_bearing': False, 'mountable': False}], 'attach': []}, 'ok': True}
init 200 {'data': {'phase': 'formatting', 'started': True}, 'ok': True}
status {'device': '/dev/sdc', 'durable_id': '', 'error': '', 'opsign': '', 'phase': 'done', 'reason': '', 'started_at': '2026-10-25T21:57:37.534911551Z', 'updated_at': '2026-10-25T21:57:40.038690849Z', 'where': '/mnt/felhom-drives/masodik'}
assign 200 {'data': {'assigned': '/mnt/felhom-drives/masodik', 'restart_required': True}, 'ok': True}
@@ -0,0 +1,10 @@
22:58:49 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH']
22:58:49 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
22:59:44 [1] deployed, controller state=running, pinned={'metube': 'ghcr.io/alexta69/metube:2026.09.29'}
deploy True
drwxrwsr-x 2 1000 1000 4096 Oct 25 19:42 /mnt/felhom-drives/adat/userdata/media/metube
67353ace24aca0fced9bb0898b2d871c4b1a799dc56bde69433b22ae8a019ac2 /mnt/felhom-drives/adat/userdata/media/metube/cc-r1283-video.mp4
tier2 200 {'ok': True, 'message': '2. mentés elindítva'}
67353ace24aca0fced9bb0898b2d871c4b1a799dc56bde69433b22ae8a019ac2 /mnt/felhom-drives/masodik/backups/secondary/metube/userdata/media/metube/cc-r1283-video.mp4
# off-site half: the rig has no off-site backup set up — not run live; governed by appbackup.tierKeeps (optional → TierSecondary only, internal/appbackup/captureset.go:191-199)
@@ -0,0 +1,6 @@
2026/10/25 21:58:11 [INFO] [sync] Initial sync: Sablonok frissítve — frissítve: homepage, metube, plex
expect n8n/docker-compose.yml ∋ N8N_DIAGNOSTICS_ENABLED=false: 1
expect opengist/docker-compose.yml ∋ 256M: 2
expect homepage/.felhom.yml ∋ family_gate: true: 1
drill main 6733764, box cache 6733764: DRILL-CONFIRMED
@@ -0,0 +1,4 @@
mirror main 6733764
cache HEAD cd0cf48
https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
@@ -0,0 +1,24 @@
# R-1285 live on the rig (BYO host Tester4-84bbe7), 2026-10-25 night
# staged: new agent 0.166.0-r1285 (main abcc1e6) installed by hand; the OLD 0.164.0 wrapper put back (no bundle-policy mode); our 0.164.0 drop-in planted (sha256 f0485da5…, sshd -T without-password)
# then ONE signed agent_config_update, test bundle 0.166.0-r1285 (sha256 e73bdb6c…)
Oct 25 14:53:46 pf-test felhom-os-apply[152336]: os-apply: BUNDLE START agent=0.166.0-r1285 sha=e73bdb6ce9fa6849 authority=signed files=32 write=1 same=26 kept=1 skipped=4
Oct 25 14:53:46 pf-test felhom-os-apply[152337]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-os-apply (replaced)
Oct 25 14:53:47 pf-test felhom-os-apply[152434]: os-apply: BUNDLE DONE agent=0.166.0-r1285 written=1 same=26 self-check=ok signers-created=False
Oct 25 14:53:47 pf-test felhom-agent[137016]: time=2026-10-25T14:53:47.830-07:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE START agent=0.166.0-r1285 sha=e73bdb6ce9fa6849 authority=signed files=32 write=1 same=26 kept=1 skipp
Oct 25 14:53:47 pf-test felhom-agent[137016]: time=2026-10-25T14:53:47.830-07:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE WROTE /usr/local/sbin/felhom-os-apply (replaced)"
Oct 25 14:53:47 pf-test felhom-agent[137016]: time=2026-10-25T14:53:47.830-07:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.166.0-r1285 written=1 same=26 self-check=ok signers-created=False"
Oct 25 14:53:47 pf-test felhom-agent[137016]: time=2026-10-25T14:53:47.830-07:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.166.0-r1285 bundle="{\"agent_version\": \"0.166.0-r1285\", \"authorit
Oct 25 14:53:47 pf-test felhom-os-apply[152438]: os-apply: BUNDLE POLICY REMOVED /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf (ours, on a BYO host — D43, R-1285)
Oct 25 14:53:48 pf-test felhom-os-apply[152447]: os-apply: BUNDLE POLICY DONE install_mode=byo removed=1
Oct 25 14:53:48 pf-test felhom-agent[137016]: time=2026-10-25T14:53:48.034-07:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE POLICY REMOVED /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf (ours, on a BYO host — D43, R-12
Oct 25 14:53:48 pf-test felhom-agent[137016]: time=2026-10-25T14:53:48.034-07:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE POLICY DONE install_mode=byo removed=1"
Oct 25 14:53:48 pf-test felhom-agent[137016]: time=2026-10-25T14:53:48.034-07:00 level=INFO msg="osupdate: bundle policy pass done" op=agent_config_update agent_version=0.166.0-r1285 bundle_policy="{\"install_mode\": \"byo\", \"removed\": [
Oct 25 14:53:48 pf-test felhom-agent[137016]: time=2026-10-25T14:53:48.913-07:00 level=WARN msg="signedjobs: signed op COMPLETED" job=14de31ae946b9f76 op=agent_config_update
total 8
drwxr-xr-x 2 root root 4096 Oct 25 14:53 .
drwxr-xr-x 4 root root 4096 Oct 25 12:21 ..
permitrootlogin yes
5
end-of-wait
[exited with code 0]
@@ -0,0 +1,22 @@
20:40:45Z [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
20:43:06Z [1] deployed, controller state=running, pinned={'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:17-alpine'}
20:43:06Z gate: recipes is gated — passed as the household (cookie set)
20:43:19Z tandoor: createsuperuser :: Running django-vite in production mode (no HMR) Superuser created successfully.
20:43:24Z tandoor: seeded superuser drill7c6d3e
20:43:24Z [4] Mentés most -> 200 {'ok': True, 'message': 'Mentés elindítva'}
20:44:22Z [5] stop 200, remove KEEPING data -> 200 {'ok': True, 'data': {'removed': 'tandoor', 'volumes_removed': ['tandoor_tandoor_media', 'tandoor_tandoor_postgres_data', 'tandoor_tandoor_static'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [],
20:44:27Z [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
20:46:33Z [1] deployed, controller state=running, pinned={'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:17-alpine'}
20:49:57Z set-aside dirs after the reinstall's backup: 0
20:49:58Z POST /backup/kept/restore -> HTTP/2 302
20:51:38Z restore: {"running": false, "op": "restore", "stack": "tandoor", "started_at": "2026-10-25T20:49:58.398659098Z", "last": {"op": "restore", "stack": "tandoor", "ok": true, "message": "A(z) tandoor: 3 adatkötet
20:52:08Z tandoor: readback of the seeded account found=True
20:52:08Z item back: True
20:52:08Z controller lines since the restore:
2026/10/25 20:51:33 [INFO] [backup] tandoor: the unit of the install the restore replaced (7dc433bf4851beb7) is set aside whole → /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T205133Z — replaced by the restored install e30f6554375a455b; no
20:52:14Z set-aside dirs after the restore: 0
20:52:14Z [4] Mentés most -> 200 {'ok': True, 'message': 'Mentés elindítva'}
20:52:47Z after the next backup — set-aside lines: 0 | set-aside dirs: 0
20:53:28Z [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'tandoor', 'volumes_removed': ['tandoor_tandoor_media', 'tandoor_tandoor_postgres_data', 'tandoor_tandoor_static'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [],
20:53:36Z after remove: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '', 'kept': '', 'subdomain': '404'}
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,9 @@
9201
pf-test
- the escrow blob in the hub, if one exists (operator UI)
[OK] UNINSTALL complete — removed: guest 9201, the felhom-agent (unit/sudoers/binary/state/config+baks/user + selfupdate-artifacts/shared-parent/mkfs-wrapper/pbs-apply-wrapper/hook-snippet/dnsmasq-snippets), the pveum role/user/token/ACL, the felhom pool, and /var/lib/felhom-install/state.json.
[INFO] golden vzdump: left in place (--remove-golden to remove).
[INFO] NOTE: the 'sudo' package was left INSTALLED (a system package, never ours to remove). dnsmasq is removed only when the install-time record says Felhom installed it; a resolver that pre-dated Felhom is never touched. The host record still exists in the hub — remove it there if desired.
UNINSTALL_RC=0
Connection to 192.168.0.150 closed.
@@ -0,0 +1,33 @@
#!/usr/bin/env python3
"""r1281household.py — R-1281's positive control on the night's rig: the household still reaches its files through
traefik after FileBrowser left traefik-public. The walk the dashboard's „Fájlok ↗" makes: GET /__files/open on the
dashboard host (with the household's session) → a one-time pass → GET /_felhom/sso on files.<domain> → the files cookie
→ the drive's listing through traefik (forwardAuth adds the secret header). No secret is printed."""
import os
import re
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import rigw # noqa: E402
w = rigw.w
w.login()
sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip()
files_host = "files." + w.DOMAIN
r = w.sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}",
f"{w.BASE}/__files/open?to=%2F"])
loc = w._loc(r.stdout)
print("1. dashboard /__files/open ->", (r.stdout or "").split("\n")[0].strip(), "| to files host:", files_host in loc)
m = re.search(r"https?://[^/]+(/_felhom/sso\?[^\s]+)", loc)
if not m:
sys.exit("no pass handed out")
r = w.sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", f"Host: {files_host}", f"{w.BASE}{m.group(1)}"])
ck = re.search(r"(?im)^set-cookie:\s*(felhom_files=[^;\r\n]+)", r.stdout or "")
print("2. files host /_felhom/sso ->", (r.stdout or "").split("\n")[0].strip(), "| files cookie set:", bool(ck))
if not ck:
sys.exit("no files cookie")
url = f"{w.BASE}/api/resources?path=%2F&" + "sou" + "rce=Adat"
r = w.sh(["curl", "-sk", "-o", "/dev/null", "-w", "%{http_code}", "-H", f"Host: {files_host}", "-H", f"Cookie: {ck.group(1)}", url])
print("3. the drive's listing through traefik with the files cookie ->", r.stdout)
r = w.sh(["curl", "-sk", "-o", "/dev/null", "-w", "%{http_code}", "-H", f"Host: {files_host}", url])
print("4. the same without any cookie (control) ->", r.stdout)
@@ -0,0 +1,58 @@
#!/usr/bin/env python3
"""r1288rig.py — R-1288 live on the night's rig (rc.2): the Tester 1 order with a throwaway Tandoor — install, back up,
remove keeping backups, reinstall (its first backup sets install A aside: one alarm), restore A's set-aside copy →
the replaced install's unit is set aside AT the restore („replaced by the restored install"), no second alarm; the
next backup sets nothing aside. Removed through the product at the end."""
import json, os, re, sys, time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import rigw # noqa: E402 (must come first: it points the walk at the rig)
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools"))
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../restore-truth-2026-10-25/tools"))
import sweep as sw # noqa: E402
import unitprobe2 as up # noqa: E402
w, fxm = sw.w, sw.fxm
APP = "tandoor"
AS = "/mnt/felhom-drives/adat/felhom-data/backups/set-aside/tandoor"
def say(*a):
print(time.strftime("%H:%M:%SZ", time.gmtime()), *a, flush=True)
w.say = say
fx = fxm.FIXTURES[APP]
sub = getattr(fx, "sub", APP)
w.login()
t0 = int(time.time())
try:
w.deploy(APP, sub) and w.wait_app(sub, "/", tries=90, delay=5)
tok = fx.seed(w, sub, say)
sw.backup_now({}, say)
sw.remove(APP, keep=True, say=say)
w.deploy(APP, sub, dict(w.GENERATED.get(APP) or {}))
for _ in range(60):
time.sleep(10)
if sw.g(f"docker logs --since {t0} felhom-controller 2>&1 | grep -c 'recovery unit of an earlier install'").strip() not in ("", "0"):
break
say("set-aside dirs after the reinstall's backup:", sw.g(f"find /mnt/felhom-drives -path '*set-aside/tandoor/*' -maxdepth 6 -mindepth 6 -type d 2>/dev/null | wc -l").strip())
html = up.page("/backups/restore")
m = re.search(r'action="/backup/kept/restore".*?name="id" value="([0-9a-f]{16})"', html, re.S)
t_r = int(time.time())
head, loc = up.post_form("/backup/kept/restore", {"id": m.group(1), "confirm": "1"})
say("POST /backup/kept/restore ->", head)
rec = {}
up.wait_restore("kept", rec)
say("restore:", rec["kept"]["status"][:200])
time.sleep(20)
say("item back:", bool(tok) and bool(fx.verify(w, sub, tok, say)))
say("controller lines since the restore:")
print(sw.g(f"docker logs --since {t_r} felhom-controller 2>&1 | grep -E 'replaced by the restored install|SET ASIDE|set aside|Event pushed: backup_run_failures' | cut -c1-260"))
say("set-aside dirs after the restore:", sw.g(f"find /mnt/felhom-drives -path '*set-aside/tandoor/*' -maxdepth 6 -mindepth 6 -type d 2>/dev/null | wc -l").strip())
t_b = int(time.time())
sw.backup_now({}, say)
say("after the next backup — set-aside lines:", sw.g(f"docker logs --since {t_b} felhom-controller 2>&1 | grep -c 'recovery unit of an earlier install'").strip(),
"| set-aside dirs:", sw.g(f"find /mnt/felhom-drives -path '*set-aside/tandoor/*' -maxdepth 6 -mindepth 6 -type d 2>/dev/null | wc -l").strip())
finally:
sw.remove(APP, keep=False, say=say)
say("after remove:", sw.leftovers(APP, sub))
+13 -11
View File
@@ -124,7 +124,7 @@ stopping line that lies.
| **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC |
| **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator |
## Apps & catalog — 16 rows (P3 6, P4 10)
## Apps & catalog — 17 rows (P3 6, P4 11)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -145,6 +145,7 @@ stopping line that lies.
| **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | WATCHING — **2026-10-22 night:** not reproducible tonight; the Immich server log of 19:38 is gone (Immich restarted at 02:16 for the off-site dump). What the household sees: the upload fails after 10 minutes and the Immich phone app sends it again. | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC |
| **R-1267** | Apps & catalog | P4 | **Every recreate of Immich leaves one empty anonymous Docker volume behind: the image declares `VOLUME /data`, the template does not map it.** Found 2026-10-24 (review round 2, while putting 9202 back): 9202 holds 296 empty anonymous volumes; each night chain's recreate of `immich-server` made a new one (12:58, 13:00, 13:01 — the three test runs; the two dangling ones this session made were removed by name, nothing pruned). The photos are NOT affected: Immich picks `/usr/src/app/upload` whenever both folders exist (`detectMediaLocation` in the image's `server/dist/services/storage.service.js:28-39`), and the template binds that folder (`templates/immich/docker-compose.yml:49`) — NOT-A-FINDING for data placement. **Seen again 2026-10-24 night on Tester 1:** privatebin's image declares `/run`, `/tmp` and `/var/lib/nginx/tmp` as volumes — each recreate (the night dump's stop/start) leaves three empty anonymous volumes; the same class as Immich's `/data` (`audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt`). | OPEN **2026-10-24 (catalog sweep):** the same class for OnlyOffice and RomM (RomM's image volume `/romm` holds only the empty mount points of its binds); after every catalog app had been removed WITH data, the sweep box still held 52 such volumes (PrivateBin 21, OnlyOffice 8, RomM 8, Immich 3, and 12 from 12:50–12:52 before the batch, holding the same nginx folders as PrivateBin's) — remove-with-data does not see them (no compose project label). All small (4–72 KB), no household data (`audits/catalog-sweep-2026-10-24/_box/anon-volumes.txt`). | — | declare the image's VOLUME paths as tmpfs or named volumes in the catalog (Immich, privatebin), or remove the dangling ones a recreate leaves, by name | CC |
| **R-1268** | Backup & restore | P3 | **Remove with „remove backups” leaves a restore's undo copy on the drive (`backups/restore-undo/<app>/`).** Seen 2026-10-24 night on Tester 1 (restore-holes Part 1, run 1): MeTube was held for support after a failed crash put-back, then removed with `remove_backups=true` — the 1.5 GB copy stayed (the removal deletes `backups/primary` and the second-drive mirror, not `restore-undo`), and nothing lists it (`audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt`; removed by hand, CC's own test data). The copy is kept on purpose while a hold exists; after the app is gone it is orphaned. | OPEN | — | Remove with `remove_backups` also deletes the app's `restore-undo` folder (under the same path guard), or lists it as a kept copy | CC |
| **R-1292** | Apps & catalog | P4 | **qBittorrent's 50 % rule reads and writes the limit of the CURRENT speed mode: with the household's alternative speed on, it sets the alternative limit, and a later switch reads as „the household changed it" — the rule stops for good.** Measured 2026-10-25 night on qBittorrent 5.2.4 (`transfer/uploadLimit`/`setUploadLimit` follow the mode; `audits/night-2026-10-25/r1261-qbittorrent-api-measured.txt`); found while fixing R-1261 (the hold now names its slot). | OPEN | — | make the rule own the NORMAL slot only (`up_limit` through `app/preferences`, as the hold does) + a test with the alternative mode on (red first); one controller release | CC |
## App updates — 5 rows (P2 1, P3 4)
@@ -176,7 +177,7 @@ stopping line that lies.
| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. | OPEN | — | one attended pass over the restore paths with these four as tests | CC |
| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
| **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
| **R-1278** | Backup & restore | P3 | **A volume app's reinstall does not offer its kept backup.** „Remove — keep my data” removes a volume app's named volumes and keeps only its backup; the kept-data question is about drive folders only (`07` §6.5, decision 36), so the reinstall starts empty and says nothing. Since D47 (2026-10-25) the earlier install's unit is set aside and listed on the restore page („Megőrzött másolatok”, „Visszaállítás”) — but the install page does not point to it (`audits/restore-truth-2026-10-25/README.md`). | OPEN | — | Design: at install, when a set-aside or removed-app unit of the app exists, say so and offer it beside a fresh start; a test | CC |
| **R-1278** | Backup & restore | P3 | **A volume app's reinstall does not offer its kept backup.** „Remove — keep my data” removes a volume app's named volumes and keeps only its backup; the kept-data question is about drive folders only (`07` §6.5, decision 36), so the reinstall starts empty and says nothing. Since D47 (2026-10-25) the earlier install's unit is set aside and listed on the restore page („Megőrzött másolatok”, „Visszaállítás”) — but the install page does not point to it (`audits/restore-truth-2026-10-25/README.md`). | **VERIFY** — fixed on main 2026-10-25 night (controller `fbec5cb`), waits for the controller release: for an app not installed, when the box keeps a backup of an earlier install (set-aside or removed-app unit), the install page says the install starts empty and links the restore page (HU and EN; render test per branch, red-proved, `redproof-r1278.txt`). Not seen live. | — | Design: at install, when a set-aside or removed-app unit of the app exists, say so and offer it beside a fresh start; a test | CC |
| **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
| **R-164** | Backup & restore | P4 | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
| **R-213** | Backup & restore | P4 | **Putting files back in place — the half the recovery screen deliberately does not do.** The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: *a screen that unlocks and then offers to overwrite is two decisions wearing one button*. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. **The operator named its requirement: a live-versus-backup comparison** — the customer must be able to see what would change before anything is overwritten | **DEFERRED** — operator ruling 2026-10-12 (S18, `09` §3 decision 213): designed after the first customers. Stays P4; re-dated: look again when the first paying household has run 30 days. **OPEN — not started, deliberately** | the comparison design (nothing exists for it yet) | After the first customers | Operator + CC |
@@ -212,9 +213,9 @@ stopping line that lies.
| **R-1232** | Backup & restore | P2 | **A reinstalled box may take its first whole-box copy to ep0 up to a week late: the old box's copy in the same household space seems to count as its last one.** Test night 2026-10-23 (`audits/night-2026-10-23/P4-testnight/`): the new Tester4 box (guest 9201) had its PBS tier applied at 19:28 UTC, but no whole-box run to ep0 came all night; the only archive in the household's ep0 space is the rehearsal box's `ct/9201` of 2026-10-20 (same guest number), and the tier's cadence is 168 h. The agent's due-check reads the newest archive for that vmid on the tier's storage (`localapi/server.go` `newestArchiveOn`) — so the old box's copy reads as „2.6 days old, not due”. If so, the ep0 copy is the OLD box's for up to a week. PLAUSIBLE — read from the code and the night's silence, not proven by a log line (the controller logs the due-check only at debug). | OPEN | — | Prove: read `GET /backup/tiers` for the box (the agent's answer per tier) on the next reinstall; fix: a tier's „last backup” counts only archives this host wrote (or after the host's enrollment); an agent test with a foreign archive in the namespace | CC |
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
| **R-1283** | Backup & restore | P3 | **MeTube's downloads are excluded from every backup — make them `optional` (D44 #25).** Catalog `c50adfc` (R-1019) marks the download folders of qBittorrent, Transmission, NZBGet, SABnzbd and MeTube `excluded` (`app-catalog-felhom.eu/templates/metube/.felhom.yml:33`). Risk: a MeTube video may be gone from its source site, so after a drive failure it cannot be fetched again. Pick: MeTube's `media/metube` becomes `optional` — the second-drive copy carries it, the off-site copy does not; NZBGet and SABnzbd stay excluded (re-downloadable). Cost: second-drive space for MeTube's folder. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §7. | OPEN | — | change the class in the catalog template, run the catalog gates (the download-binds gate and its decoys must accept MeTube as `optional`), check on a Tier-0 box that a second-drive copy carries the folder and the off-site run does not | CC |
| **R-1286** | Backup & restore | P4 | **demo-hp's kimai has been HELD since 2026-10-14 09:06Z („held after a failed restore whose rollback also failed") and no row names it.** Seen 2026-10-25 by the D47 upgrade count (attended release run): kimai is the one app of 12 on demo-hp without an install id, because the start-up step skips a held app; `app.yaml` says deployed, no container exists, boot reconciliation leaves it held, the night update leg has skipped it since at least 2026-10-16 (`audits/burndown-apps-2026-10-16/P1/night-chain-log.txt`). A test box; the hold is the designed fail-closed state (R-379), but 11 days with no record is how a held household app would go unseen too. | OPEN | — | read why the 2026-10-14 restore and its rollback failed (demo-hp's controller log of that day, if kept; the hold record); then clear the hold (`--clear-restore-hold kimai` + a restart) or remove the app through the product; check that a held app raises a standing operator signal, else file that | CC |
| **R-1288** | Backup & restore | P4 | **After the household restores an earlier install's set-aside copy, the next backup sets the REINSTALL's own unit aside too — a second kept copy on the restore page and a second operator alarm.** Seen 2026-10-25 on Tester 1 (0.343.0, Tandoor, attended release run): reinstall → its first backup set install A's unit aside (17:31:55) and wrote install B's unit; restoring A's copy makes the app install A again („the restored data belongs to install 795457b5… — its backups follow it (D47)"); the next 20-minute backup step (17:36:54) found B's unit in `primary/` and set it aside. Nothing is lost; the second copy is the fresh reinstall's near-empty data. D47's design does what it says; the question is whether a restore of a set-aside unit should retire the unit it replaces instead of leaving it for the next writer. | OPEN | — | read `07` §8.2d/§8.2e for the intended end state; if the restore should retire B's unit: move it aside AT the restore (one line, no alarm, „replaced by the restored install") with a test; else document the second copy and word its alarm (R-1287) | CC |
| **R-1283** | Backup & restore | P3 | **MeTube's downloads are excluded from every backup — make them `optional` (D44 #25).** Catalog `c50adfc` (R-1019) marks the download folders of qBittorrent, Transmission, NZBGet, SABnzbd and MeTube `excluded` (`app-catalog-felhom.eu/templates/metube/.felhom.yml:33`). Risk: a MeTube video may be gone from its source site, so after a drive failure it cannot be fetched again. Pick: MeTube's `media/metube` becomes `optional` — the second-drive copy carries it, the off-site copy does not; NZBGet and SABnzbd stay excluded (re-downloadable). Cost: second-drive space for MeTube's folder. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §7. | **VERIFY** — built in the DRILL catalog 2026-10-25 night (`app-catalog-drill` `6733764`), waits for the operator's catalog push: MeTube `optional`; the download-binds gate allows it by a named ruling only (red-proved, `redproof-r1283.txt`). **Live on the rig (drill `6733764`):** a 3 MB file in MeTube's folder → „Tier 2 copied metube … 1 leg(s)", the same sha256 on the second drive (`rig/r1283-live.txt`). The off-site half NOT live (the rig has no off-site): it is `appbackup.tierKeeps` (optional → the second drive only). **Earlier state:** OPEN | — | change the class in the catalog template, run the catalog gates (the download-binds gate and its decoys must accept MeTube as `optional`), check on a Tier-0 box that a second-drive copy carries the folder and the off-site run does not | CC |
| **R-1286** | Backup & restore | P4 | **demo-hp's kimai has been HELD since 2026-10-14 09:06Z („held after a failed restore whose rollback also failed") and no row names it.** Seen 2026-10-25 by the D47 upgrade count (attended release run): kimai is the one app of 12 on demo-hp without an install id, because the start-up step skips a held app; `app.yaml` says deployed, no container exists, boot reconciliation leaves it held, the night update leg has skipped it since at least 2026-10-16 (`audits/burndown-apps-2026-10-16/P1/night-chain-log.txt`). A test box; the hold is the designed fail-closed state (R-379), but 11 days with no record is how a held household app would go unseen too. | **WAITING-ON-OPERATOR** — read 2026-10-25 night: the hold is NOT a failed restore — `reason: unhealthy_stop`, `crash_loop`, 2026-10-14 09:06:31Z, during the night power drill on demo-hp (8 real cuts); every surface misnamed it — fixed on main (controller `838c834`, `holdWhy`, red-proved). A held app gives the operator no lasting signal → R-1291. **The pick (STATUS):** press Start (the household's Start lifts this hold; a new crash loop holds it again) or remove Kimai through the product. **Earlier state:** OPEN | — | read why the 2026-10-14 restore and its rollback failed (demo-hp's controller log of that day, if kept; the hold record); then clear the hold (`--clear-restore-hold kimai` + a restart) or remove the app through the product; check that a held app raises a standing operator signal, else file that | CC |
| **R-1288** | Backup & restore | P4 | **After the household restores an earlier install's set-aside copy, the next backup sets the REINSTALL's own unit aside too — a second kept copy on the restore page and a second operator alarm.** Seen 2026-10-25 on Tester 1 (0.343.0, Tandoor, attended release run): reinstall → its first backup set install A's unit aside (17:31:55) and wrote install B's unit; restoring A's copy makes the app install A again („the restored data belongs to install 795457b5… — its backups follow it (D47)"); the next 20-minute backup step (17:36:54) found B's unit in `primary/` and set it aside. Nothing is lost; the second copy is the fresh reinstall's near-empty data. D47's design does what it says; the question is whether a restore of a set-aside unit should retire the unit it replaces instead of leaving it for the next writer. | **VERIFY** — fixed on main 2026-10-25 night (controller `b5863a9`; the brief's safest reading, `07` §8.2d), waits for the controller release: the replaced install's unit set aside AT the restore, no alarm. **Live on the rig (rc.2):** the line, the account back, the next backup set nothing aside, one set-aside mail only (`rig/r1288-live.txt`). **Earlier state:** OPEN | — | read `07` §8.2d/§8.2e for the intended end state; if the restore should retire B's unit: move it aside AT the restore (one line, no alarm, „replaced by the restored install") with a test; else document the second copy and word its alarm (R-1287) | CC |
## Storage & devices — 14 rows (P3 5, P4 9)
@@ -233,7 +234,7 @@ stopping line that lies.
| **R-1213** | Storage & devices | P4 | **Operator detail still on two household pages:** the registered drive card's `ext4 · /dev/sdb1 · Model` line (`templates/storage.html`; W14 hid only the agent drive cards), and the heading „Távoli mentés (3. mentés) — titkosított, offsite" with the English „offsite". Found by the night helper 2026-10-21 (W11/W14 follow-up). | OPEN | — | Move the line behind „Részletek"; „…titkosított, házon kívüli" | CC |
| **R-1275** | Storage & devices | P4 | **An English household sees the drive names in Hungarian** on the install page (and wherever a storage label shows): „Belső tárhely — 52.0 GB free (default)”, „Külső tárhely (hdd_1)” (catalog sweep 2026-10-24, `audits/catalog-sweep-2026-10-24/navidrome/page-install-en.txt`). The label is stored once, in Hungarian, when the drive is found (`felhom-controller/controller/internal/settings/settings.go` `InferStorageLabel`); a household may rename it. | OPEN | — | Render a label that still equals the inferred default in the reader's language; a test per language | CC |
| **R-1276** | Storage & devices | P4 | **„Remove with data” leaves the app's empty parent folder** (`<drive>/appdata/<app>`) when the template binds a subfolder: Grimmory, RomM, Paperless-ngx (catalog sweep 2026-10-24; 4 KB each; the bound folders were removed). Harmless (empty, not counted as old data), untidy for a household that browses the drive. | OPEN | — | Remove the parent when it is empty after the bound folders go; a test | CC |
| **R-1282** | Storage & devices | P3 | **The „Később" 03:30 media restart is forgotten by a controller restart — store the due time (D44 #24).** The restart is an in-memory timer (`felhom-controller/controller/internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`); after a controller restart only the notice stays. Risk: media apps do not see the new drives until the household presses „now". Pick: store the due time and run the restart after a controller restart (a due time already past runs at once, within the same rules). Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §11. | OPEN | — | a small controller change + a test that a stored due time survives a restart and fires once; rides a controller release | CC |
| **R-1282** | Storage & devices | P3 | **The „Később" 03:30 media restart is forgotten by a controller restart — store the due time (D44 #24).** The restart is an in-memory timer (`felhom-controller/controller/internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`); after a controller restart only the notice stays. Risk: media apps do not see the new drives until the household presses „now". Pick: store the due time and run the restart after a controller restart (a due time already past runs at once, within the same rules). Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §11. | **VERIFY** — fixed on main 2026-10-25 night (controller `b5863a9`), waits for the controller release: the „later" time stored, resumed at start, once (`r1282_media_later_test.go`, red-proved). **Earlier state:** OPEN | — | a small controller change + a test that a stored due time survives a restart and fires once; rides a controller release | CC |
## Security & access — 26 rows (P1 2, P2 4, P3 14, P4 6)
@@ -263,8 +264,8 @@ stopping line that lies.
| **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f <conf>` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC |
| **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC |
| **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator |
| **R-1280** | Security & access | P3 | **A signed operation's window has no maximum length — cap it at 24 h (D44 #22).** The agent (`felhom-agent/internal/authz/verifier.go`) and the root wrapper (`configs/felhom-os-apply`) check `now ∈ [issued_at, expires_at]` with a 2-minute skew on `issued_at`, but neither checks `expires_at − issued_at`; `felhom-opsign` defaults to 30 min (`cmd/felhom-opsign/main.go:57`) and the nonce store keeps a record until its expiry. Risk: a signed job made with a long window by mistake stays usable for that whole time. Pick: the agent and the root wrapper refuse a window longer than 24 h. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/04-control-plane-authorization.md` §2. | OPEN | — | one agent change in both checkers + a test that a 24 h + 1 s window is refused (red first) and a 24 h one is accepted; rides an agent release (agent releases stop for the operator's vouch) | CC |
| **R-1281** | Security & access | P2 | **Every app container can reach FileBrowser directly; the only guard is the secret header name — take FileBrowser off the app networks (D44 #23).** FileBrowser trusts its proxy header (`auth.methods.proxy`); every app container can reach `filebrowser:80` without traefik; the guard is that the header NAME is a per-box secret that traefik's forwardAuth strips from client requests (`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`). Risk: an app that learns the name (a hacked app on the box) reads and writes every household file. Pick: FileBrowser joins only the network traefik uses, so no app container reaches it directly. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/13-threat-model.md` §5. | OPEN | — | a compose/network change in the controller's infra render + a test on the rendered networks; a live check on a Tier-0 box that a forged header from an app container no longer reaches FileBrowser (the 2026-10-18 measurement, now refused) while the dashboard's file manager still signs in; one controller release | CC |
| **R-1280** | Security & access | P3 | **A signed operation's window has no maximum length — cap it at 24 h (D44 #22).** The agent (`felhom-agent/internal/authz/verifier.go`) and the root wrapper (`configs/felhom-os-apply`) check `now ∈ [issued_at, expires_at]` with a 2-minute skew on `issued_at`, but neither checks `expires_at − issued_at`; `felhom-opsign` defaults to 30 min (`cmd/felhom-opsign/main.go:57`) and the nonce store keeps a record until its expiry. Risk: a signed job made with a long window by mistake stays usable for that whole time. Pick: the agent and the root wrapper refuse a window longer than 24 h. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/04-control-plane-authorization.md` §2. | **VERIFY** — fixed on main 2026-10-25 night (agent `d912307`), waits for the agent release + the operator's vouch: 24 h in the verifier, the root wrapper and felhom-opsign (red-proved, `redproof-r1280.txt`). **Earlier state:** OPEN | — | one agent change in both checkers + a test that a 24 h + 1 s window is refused (red first) and a 24 h one is accepted; rides an agent release (agent releases stop for the operator's vouch) | CC |
| **R-1281** | Security & access | P2 | **Every app container can reach FileBrowser directly; the only guard is the secret header name — take FileBrowser off the app networks (D44 #23).** FileBrowser trusts its proxy header (`auth.methods.proxy`); every app container can reach `filebrowser:80` without traefik; the guard is that the header NAME is a per-box secret that traefik's forwardAuth strips from client requests (`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`). Risk: an app that learns the name (a hacked app on the box) reads and writes every household file. Pick: FileBrowser joins only the network traefik uses, so no app container reaches it directly. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/13-threat-model.md` §5. | **VERIFY** — fixed on main 2026-10-25 night (controller `7ea942a`), waits for the controller release: FileBrowser served on `felhom-files` (traefik + OnlyOffice while linked), off traefik-public. **Live on the rig (rc.3):** before, both app containers signed in with the forged header (200); after, no answer (`000`); the household's „Fájlok ↗" walk 200, no cookie 401 (`rig/r1281-*.txt`). The migration live too: FileBrowser put back on traefik-public by hand → the next 5-minute round logged „left traefik-public — no app container reaches it any more" and the apps got no answer again. OnlyOffice's join by test only (not installed on the rig). On the release every box re-creates FileBrowser once (its compose comment changed). **Earlier state:** OPEN | — | a compose/network change in the controller's infra render + a test on the rendered networks; a live check on a Tier-0 box that a forged header from an app container no longer reaches FileBrowser (the 2026-10-18 measurement, now refused) while the dashboard's file manager still signs in; one controller release | CC |
## Box system & updates — 13 rows (P2 1, P3 7, P4 5)
@@ -282,14 +283,14 @@ stopping line that lies.
| **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
## Monitoring & notifications — 23 rows (P2 1, P3 16, P4 6)
## Monitoring & notifications — 24 rows (P2 1, P3 17, P4 6)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | OPEN — **now in controller 0.343.0 (released 2026-10-25); the three edges act only on a box whose agent is ≥ 0.165.0 (it alone reports `waiting_for_quiet`) — fix before any household gets agent 0.165.0 by signed update** | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC |
| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | **VERIFY** — fixed on main 2026-10-25 night (controller `a79a292`), waits for the controller release (before any household gets agent 0.165.0 by signed update): every running seeder held, a removed app's record dropped, qBittorrent's slot named (`up_limit`/`alt_up_limit`). Red-proofs `audits/night-2026-10-25/redproof-r1261.txt`. **Live on the night's rig** (BYO VM, agent 0.165.0, rc.1, two seeders): both held, 94.8 Mbit/s, both limits back; a speed-mode switch mid-test left no 1 KiB/s (`rig/hold-watch*.txt`); the removed-app case by test only. **Earlier state:** OPEN — **now in controller 0.343.0 (released 2026-10-25); the three edges act only on a box whose agent is ≥ 0.165.0 (it alone reports `waiting_for_quiet`) — fix before any household gets agent 0.165.0 by signed update** | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC |
| **R-1262** | Monitoring & notifications | P3 | **The upload watch treats ANY viewer as an explanation for any amount of upload, so a hacked Plex can send freely while one family member watches.** Found 2026-10-24 (SEC-1 X1): `felhom-controller/controller/internal/upwatch/watch.go:275,299`; `08` §9 states the rule, its limits list did not name this case. | OPEN | — | propose a cap (viewers × a bitrate ceiling) for the operator's R-1244 design; `08`'s limits list names the case meanwhile | CC |
| **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
@@ -312,6 +313,7 @@ stopping line that lies.
| **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
| **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
| **R-1287** | Monitoring & notifications | P3 | **A D47 set-aside mails the operator as „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" — no backup failed; the body says the right thing („its backup belonged to an earlier install … SET ASIDE whole").** Seen 2026-10-25 on Tester 1 with controller 0.343.0 (attended release run): two such mails at 17:31:56Z and 17:36:55Z. The controller sends the set-aside as `backup_run_failures` leg `unit-set-aside`, `Failed: 1` (so the hub never drops a line that claims zero); the hub's subject for that event is fixed („N of M apps failed to back up", `hub/internal/notify/templates.go:353`). The CLAUDE.md corollary: an alarm's text must say what it counts. After the floor, every household reinstall mails the operator a red „failed to back up". | **VERIFY** — fixed on main 2026-10-25 night, waits for the hub release (v0.171.0): a set-aside-only digest is a warning with „… set aside — nothing failed (<apps>)"; a run with a real failure stays red (`hub/internal/notify/r1287_setaside_test.go`, red-proved) | — | two-repo: the hub words a `unit-set-aside` leg on its own („… a backup of an earlier install was set aside — nothing failed", warning, not error) with a test on the rendered subject; or a new event type through the allow-list (`hub-event-allowlist-gotcha`); one hub release | CC |
| **R-1291** | Monitoring & notifications | P3 | **A held app gives the operator no lasting signal: one event at the stop, then nothing — demo-hp's Kimai sat held 11 days unseen.** Found 2026-10-25 night (R-1286). No held-app state reaches the hub (the report carries none; the hub shows none); the crash-loop stop sends `app_stopped_unhealthy` once. On a household box an app held after a crash loop, a failed update or a failed restore stays stopped until someone opens its page. | OPEN | — | a two-repo design: the controller reports its holds (stack, reason, since), the hub shows them on the host and mails a reminder after N days (the D-level of the reason); one row's worth of design first | CC |
## Hub & operator — 13 rows (P3 5, P4 8)
+4
View File
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-10-26 (morning)
**Updated 2026-10-25 (evening, attended release).** Every box runs the new controller: a restore that does not start says so and keeps both copies (D49), a reinstall never touches the old backup, a restore downloads first. The hub stores the recovery package in one step; the new agent is vouched; the catalog has three fixes.
## Moved from STATUS on 2026-10-25 (attended release run)
**Updated 2026-10-25 (Sunday).** A reinstall keeps old backups whole; a restore downloads first and tells the truth.