Night 2026-10-25 Part 4–5: each fix's [DESIGN] line (04, 05, 07, 08, 11, 13, 15), the register (R-1261, R-1281, R-1282, R-1288, R-1280, R-1285, R-1283, R-1287, R-1289, R-1278 to VERIFY; R-1286 waiting on the operator; R-1291 and R-1292 opened), STATUS (D51 Kimai, the Monday list), the night report with its three-layer teardown, and the rig evidence (scanned: 0 known-secret hits)
gates / gates (push) Successful in 13m42s
gates / gates (push) Successful in 13m42s
This commit is contained in:
1 parent
f63284861e
commit
9258cdd11c
33 files changed
+57024
-22
No files matched your search
+14
@@ -16,6 +16,20 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-25 night (unattended, commit only) — the night's fixes, each red-proved; proven live on a BYO test VM (demo-hp
|
||||
> VM 9372, customer Tester4, agent 0.165.0, controller rc.1–rc.3).** Controller (`a79a292`…`7ea942a`, rides v0.344.0):
|
||||
> R-1261 the line hold holds EVERY running seeder, drops a removed app's record, names qBittorrent's slot (`up_limit` /
|
||||
> `alt_up_limit` via `app/preferences` — `transfer/uploadLimit` follows the current speed mode, measured); R-1288 the
|
||||
> restore of a set-aside unit sets the replaced unit aside AT the restore, no alarm (the brief's safest reading, `07`
|
||||
> §8.2d changed); R-1282 the „later" media restart stored and resumed; R-1281 FileBrowser served on its own
|
||||
> `felhom-files` network (traefik + OnlyOffice while linked), off traefik-public — the forged header from an app went
|
||||
> 200 → no answer; R-1286 every hold reason says what it is (Kimai's was a crash-loop stop, not a failed restore). Hub
|
||||
> (`f6328486`, v0.171.0): R-1287 a set-aside-only digest is a warning „nothing failed"; R-1289 suffixed versions never
|
||||
> „the fleet's newest". Agent (`d912307`, `abcc1e6`, next agent release + vouch): R-1280 a signed window ≤ 24 h; R-1285
|
||||
> the agent runs the wrapper's `bundle-policy` pass after every bundle install. Drill catalog `6733764`: R-1283 MeTube
|
||||
> `optional` (a named ruling in the download-binds gate). New rows R-1291 (a held app has no lasting signal), R-1292
|
||||
> (the 50 % rule follows the speed mode).
|
||||
|
||||
> **2026-10-25 (evening, attended) — D50 = A; installer 1.36.1 published.** (`09` §3 decision 262.) The registry keeps the
|
||||
> newest 5 agent versions + the MinAgent floor + the day-0 agent; the agent CI asserts exactly these
|
||||
> (`retention-policy.json` `pinned`, copied from hub state by the vouching session — CI holds no hub secret). The operator
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
# REPORT — installer 1.36.1 published; the night's fixes (2026-10-25 evening → 2026-10-26 morning)
|
||||
|
||||
**Plain words first.**
|
||||
1. **Every package that is still named is there.** You pruned old controller, agent and golden packages by hand; nothing a
|
||||
box or the installer can still ask for is missing (the agent 0.165.0 + bundle, the 0.158.1 floor agent, golden
|
||||
0.331.0, the agent tags from 0.158.1 up, controller 0.343.0 / 0.342.0 / 0.331.0, samba 1.2.0, the ISO).
|
||||
2. **Installer 1.36.1 is live, and its three messages were seen:** a wrong customer → „not found (404)", a wrong
|
||||
passphrase → „passphrase REJECTED (401)", a customer whose box is live → „already has a live box (409)" (R-1167's
|
||||
sentence, which could never show before).
|
||||
3. **The agent CI is green with D50 = A** (the newest 5 versions + the MinAgent floor + the day-0 agent).
|
||||
4. **R-1261 is fixed and proven before Tuesday** — on a test VM with two seeders: both held, 94.8 Mbit/s measured, both
|
||||
limits back; a speed-mode switch mid-test never left 1 KiB/s. It reaches households with Monday's controller release.
|
||||
5. **You decide:** Kimai on demo-hp — press Start, or remove it (STATUS, D51). And the Monday release list (STATUS).
|
||||
|
||||
## Each fix, before → after
|
||||
|
||||
- **R-1261 (controller `a79a292`):** one seeder held, a removed app's record kept forever, qBittorrent's limit put back
|
||||
in whatever slot its mode showed → every seeder held, the record dropped, the held slot named. Live on the rig.
|
||||
- **R-1287 (hub `f6328486`, v0.171.0):** a set-aside mailed „🔴 1 of 1 apps failed to back up" → „⚠️ … a backup of an
|
||||
earlier install was set aside — nothing failed", a warning. Test on the real dispatch path.
|
||||
- **R-1288 (controller `b5863a9`):** a set-aside restore's replaced unit was set aside at the NEXT backup, with a second
|
||||
alarm → set aside at the restore, no alarm. Live on the rig (one set-aside mail only).
|
||||
- **R-1281 (controller `7ea942a`):** an app container signed in to FileBrowser with a forged header (200) → no answer;
|
||||
the household's file manager still 200; a FileBrowser left on traefik-public is moved off by the next round. Live.
|
||||
- **R-1289 (hub `f6328486`):** a dev build no box ran was „the fleet's newest" → test builds never count.
|
||||
- **R-1282 (controller `b5863a9`):** the 03:30 media restart was forgotten by a controller restart → stored, resumed once.
|
||||
- **R-1280 (agent `d912307`):** a signed window of any length → at most 24 h (verifier, root wrapper, felhom-opsign).
|
||||
- **R-1285 (agent `abcc1e6`):** a BYO host lost our SSH file one update late → the agent runs the new wrapper's policy
|
||||
pass after every bundle install.
|
||||
- **R-1283 (drill catalog `6733764`):** MeTube's downloads `excluded` → `optional` (a named ruling in the gate).
|
||||
- **R-1286 (controller `838c834`):** Kimai's crash-loop hold read „a failed restore whose rollback also failed" → every
|
||||
hold reason says what it is. The cause: the crash-loop guard during the 10-14 power drill.
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
- **Machine (rig VM 9372 on demo-hp):** the product's own `--uninstall` ran first (rc 0, „UNINSTALL complete — removed:
|
||||
guest 9201, the felhom-agent …", `audits/night-2026-10-25/rig/uninstall-console.txt`); the host journal was copied off
|
||||
before (`rig/rig-host-journal.txt`). Then `qm destroy 9372 --purge 1`: `qm list` shows no 9372, its folder on
|
||||
`nvme-scratch` is gone, the plain-PVE ISO is removed from demo-hp, the ISO work folder on DooPlex is removed, the old
|
||||
host key is removed from `known_hosts`.
|
||||
- **Host (demo-hp):** guest 9202's test tracker, downloader container, test files and image are removed (0 containers,
|
||||
0 files, port 6969 closed). `nvme-scratch` is a standing storage (it holds 341, 9202, 9401) — this session did not add
|
||||
it, so it stays. `local-lvm` (the thin pool) was never used: 48 %.
|
||||
- **Hub:** host `Tester4-84bbe7` stays on the Hosts page — a host delete is the operator's (STATUS, „Also Monday"). The
|
||||
R-1285 test package was deleted from the registry (204, then 404).
|
||||
- **Secrets:** the scratch session, CSRF and header files, the dashboard password file, the rig root password file, the
|
||||
test bundle and the test agent build are shredded. The evidence was scanned against the four known values (through
|
||||
`sec`) and four token shapes: 0 hits (one shape hit is code that reads a file). The `sec` entries
|
||||
`RIG_2026_10_25_DASH_PW` and `RIG_2026_10_25_SETUP_CODE` remain; `sec` has no delete, and the box they opened is gone.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
|
||||
|
||||
**Updated 2026-10-25 (evening, attended release).** Every box runs the new controller: a restore that does not start says so and keeps both copies (D49), a reinstall never touches the old backup, a restore downloads first. The hub stores the recovery package in one step; the new agent is vouched; the catalog has three fixes.
|
||||
**Updated 2026-10-26 (morning).** Installer 1.36.1 is live: a wrong passphrase now says so. Overnight, commit only: the torrent line test holds every seeder; FileBrowser is off the app network; a set-aside mails as a set-aside; the four D44 picks are built and proven on a test VM.
|
||||
|
||||
## What works
|
||||
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
|
||||
@@ -31,6 +31,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
|
||||
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
||||
|---|---|---|---|---|---|---|
|
||||
| D51 | R-1286 | demo-hp's Kimai was stopped by the crash-loop guard on 10-14 (the power drill) and is still held. Start it again, or remove it? | **Press Start** (it lifts this kind of hold; if it crash-loops again it is held again and we look) | remove it through the product: the test app and its data are gone | it stays stopped; nothing else waits on it | — |
|
||||
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
|
||||
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
|
||||
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
|
||||
@@ -47,7 +48,8 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
|
||||
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
|
||||
|
||||
**Day, 2026-10-23 — your do list.**
|
||||
**Monday 2026-10-26 (attended; each a STOP).** 1. Hub v0.171.0 (R-1287, R-1289). 2. Agent v0.166.0 + your vouch (R-1280, R-1285). 3. Controller v0.344.0 — before any household gets agent 0.165.0 (R-1261, R-1281, R-1288, R-1282, R-1286, R-1278). 4. Catalog: MeTube (R-1283). 5. Dated: R-1183 today, R-1250 Tuesday. 6. Hub Hosts page: delete the dead `Tester4-84bbe7`. If not: the fixes stay unreleased; the host alarms.
|
||||
**2026-10-23 do list.**
|
||||
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
|
||||
2. On the hub's Hosts page: Discard appliance 49. If not: a dead entry alarms.
|
||||
3. Give the homelab session R-1241, R-1242, the lists at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` and `documentation/audits/dooplex-alarms-2026-10-24/README.md`, and an upload alarm on DooPlex's `eno1` (~100 Mbit/s line; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups.
|
||||
|
||||
@@ -80,7 +80,9 @@ nonce (DoS-safe). Each layer is mandatory and was proven to reject independently
|
||||
the root wrapper (`configs/felhom-os-apply`) alike (corrected 2026-10-24). **[DESIGN — the operator picked a code change
|
||||
2026-10-25 (D44); row R-1280]:** neither caps the window's LENGTH (`expires_at − issued_at`); `felhom-opsign` defaults to 30 min
|
||||
(`cmd/felhom-opsign/main.go:57`), and the nonce store keeps a record until its expiry. The pick: the agent and the
|
||||
root wrapper refuse a window longer than 24 h. Not built yet;
|
||||
root wrapper refuse a window longer than 24 h. **Built 2026-10-25 night (agent `d912307`, unreleased — rides the next
|
||||
agent release):** `authz.MaxWindow = 24h` (`ErrWindowTooLong`), the wrapper's `MAX_WINDOW_S`, and `felhom-opsign -ttl`
|
||||
refuses more; exactly 24 h is accepted;
|
||||
- **nonce** — unseen within the window (the nonce store **must be persistent across agent restarts**
|
||||
and expiry-pruned; a non-persistent store reopens the replay window after a restart).
|
||||
|
||||
|
||||
@@ -139,7 +139,9 @@ walk is green; the golden is baked weekly and vouched by the operator; a new box
|
||||
itself to the floor (R-1052: not yet proven end to end). Setting the global floor today needs the operator page with the
|
||||
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
|
||||
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
|
||||
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
|
||||
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **„The fleet" is the
|
||||
newest RELEASED controller any box reported** — a test build (`-rc.N`, `-dev…`) never counts (R-1289, hub v0.171.0,
|
||||
unreleased: the semver compare could not read a suffix and kept a dev build no box ran as the fleet's newest). **The agent list
|
||||
offers real agent packages only (R-1011, hub 0.162.0, deployed since; corrected 2026-10-24):** the sha lookup asks for exactly the
|
||||
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
|
||||
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
|
||||
|
||||
@@ -1720,7 +1720,7 @@ names the app and both identities). Nothing moves a unit while that app is being
|
||||
telepítés mentése") with **„Visszaállítás"** — the one unit-restore body, with the unit's own database password — and
|
||||
„Eltávolítás". Only those two, and the app's remove with „delete its backups too", ever remove it (the §8.2a / R-1252 H9
|
||||
rule). After a restore from it, the app is that unit's install; the app's own unit of the install it replaced is then
|
||||
set aside at the next backup — never lost.
|
||||
set aside ~~at the next backup~~ **at the restore itself** — never lost. **[DESIGN — R-1288, 2026-10-25 night, the brief's safest reading; controller `b5863a9`, unreleased]:** the restore of a set-aside unit moves the replaced install's unit aside whole right after it (success, or „not started yet" — the data stays, D49 = A), one log line „replaced by the restored install", **no operator alarm** — the household chose this restore; before, the next backup's writer set it aside with a second alarm and a second kept copy (measured on Tester 1, 0.343.0). **[FACT]** on the night's rig (rc.2): the line, the account back, the next backup set nothing aside, one set-aside mail only (`audits/night-2026-10-25/rig/r1288-live.txt`). A set-aside's operator mail (R-1287, hub v0.171.0, unreleased) reads „⚠️ … a backup of an earlier install was set aside — nothing failed", a warning.
|
||||
|
||||
**Known limit, stated.** A box that reinstalled and refreshed BEFORE this release already holds a rewritten unit; its
|
||||
old password is gone and nothing can bring it back. A unit from before D47 of an app with no carried secret has no
|
||||
@@ -1728,7 +1728,7 @@ signal: it is treated as this install's (the old behaviour) and gets the identit
|
||||
**Why a volume app's reinstall asks no kept-data question:** the question (§6.5, decision 36) is about the app's drive
|
||||
FOLDER (`${HDD_PATH}` binds under `appdata/`); a volume app keeps nothing there — „remove, keep my data" removes its named
|
||||
volumes and keeps only its backups — so the install starts empty, and the kept backup is now the set-aside unit on the
|
||||
restore page. (The missing offer at install is its own row, R-1278.)
|
||||
restore page. **[DESIGN — R-1278, controller `fbec5cb`, unreleased]:** the install page now says so for an app not installed — „the new install starts empty; load the backup afterwards on the Restore page" — with a link (set-aside and removed-app units counted).
|
||||
|
||||
### 8.2e A restore whose app does not start is never „success" [DESIGN, R-1270, 2026-10-25 — controller `b680db8` + fix A; D49 = A (operator 2026-10-25); released in controller 0.343.0]
|
||||
|
||||
|
||||
@@ -591,6 +591,15 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me
|
||||
start-up check puts it back; the 50 % rule waits while a record exists. Measured on demo-hp with a real seed (10
|
||||
Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; with it the uplink fell to 0.02–0.16 Mbit/s,
|
||||
94.2 Mbit/s measured, the app held 56.8 s and back to 1 249 280 B/s exactly (`audits/dooplex-alarms-2026-10-24/`).
|
||||
**[DESIGN — R-1261, 2026-10-25 night; controller `a79a292`, unreleased]:** EVERY running seeder is held (qBittorrent
|
||||
and Transmission — one left seeding kept the test refused `not-quiet` every time); all records first
|
||||
(`upwatch.line_test_holds`), then the apps; a record whose app was removed while held is dropped (it used to stop the
|
||||
50 % rule for good); qBittorrent's hold names its slot (`up_limit` / `alt_up_limit` through `app/preferences` —
|
||||
`transfer/uploadLimit` follows the CURRENT speed mode, measured on 5.2.4), so the household's switch to the
|
||||
alternative speed during the test never leaves 1 KiB/s behind. **[FACT]** the night's rig (BYO VM, agent 0.165.0,
|
||||
rc.1, two seeders to 9202's qBittorrent): with a mode switch mid-test both limits came back, the test refused
|
||||
(the switch let qBittorrent upload); without, both held (qBittorrent in its alternative slot), 94.8 Mbit/s measured,
|
||||
both limits back exactly (`audits/night-2026-10-25/rig/hold-watch*.txt`).
|
||||
While the box owns it, qBittorrent's page says: „A qBittorrent
|
||||
feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
|
||||
6. **Limits, stated:** *(added 2026-10-24, review round 2: ANY viewer explains any amount of upload, so a hacked app can send freely while one real playback runs — `upwatch/watch.go:275,299`; R-1262)*; an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
|
||||
|
||||
@@ -79,7 +79,11 @@ boxes installed. Nobody keeps a hand-written list.
|
||||
**[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased]** The host's root-key-only sshd
|
||||
drop-in (D4, R-982) is an APPLIANCE-only file of the config bundle: written only when the root-owned install record
|
||||
says `appliance`; on a `byo` record nothing is written and a copy the bundle wrote earlier (this path, our first line)
|
||||
is removed; a box whose record proves nothing is left as it is (R-1257).
|
||||
is removed; a box whose record proves nothing is left as it is (R-1257). **[FACT] 2026-10-25 (BYO test host, agent
|
||||
0.164.0 → 0.165.0):** the removal came one bundle LATE — the bundle that brings the new wrapper is planned by the OLD
|
||||
one (R-1285). **[DESIGN — R-1285, agent `abcc1e6`, unreleased]:** right after every bundle install the agent asks the
|
||||
wrapper now on disk for a policy pass (`mode: "bundle-policy"`), which removes our appliance-only files from a BYO
|
||||
record (no signature: it only removes a file we wrote; a failing `sshd -t` puts it back).
|
||||
- **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5).
|
||||
- **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in
|
||||
the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since
|
||||
|
||||
@@ -84,7 +84,13 @@ directly; the only guard is that the header NAME is a per-box secret, which trae
|
||||
client request and sets only for a browser holding the dashboard's files pass
|
||||
(`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`, `internal/web/files_sso.go`). The name file's
|
||||
creation race was fixed by review round 2 (controller `b01bb4b`). The pick: FileBrowser joins only the network traefik uses, so no app
|
||||
container reaches it directly. Not built yet.
|
||||
container reaches it directly. **Built 2026-10-25 night (controller `7ea942a`, unreleased):** a network of its own for
|
||||
the route, `felhom-files` — traefik is on it, the CHECKED FileBrowser joins it by id after R-1126's default-login check,
|
||||
and FileBrowser leaves traefik-public; OnlyOffice joins it while — and only while — its FileBrowser link is on (it
|
||||
fetches and saves documents at `http://filebrowser:80`), the one stated exception. **[FACT] measured on the night's rig:**
|
||||
before, from the qBittorrent and Transmission containers the forged header as the household user got the drive's
|
||||
listing (200); after, no answer (`000`); the household's „Fájlok ↗" walk still gets 200 through traefik
|
||||
(`audits/night-2026-10-25/rig/r1281-*.txt`).
|
||||
|
||||
## 6. What we accept (do not re-report)
|
||||
|
||||
|
||||
@@ -169,8 +169,10 @@ download (R-1019). *(Fixed since: catalog `c50adfc` marks the download and watch
|
||||
`app-catalog-felhom.eu/templates/transmission/.felhom.yml:46-51`; noted 2026-10-24.)* **[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1283]:** the same commit also
|
||||
marks the download folders of NZBGet, SABnzbd and MeTube `excluded` (`templates/{nzbget,sabnzbd,metube}/.felhom.yml`).
|
||||
NZBGet and SABnzbd stay excluded (their files can be fetched again). The pick for MeTube: `optional`, so the
|
||||
second-drive copy carries it and the off-site copy does not — a MeTube video can be gone from its source site. Not
|
||||
built yet. **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
|
||||
second-drive copy carries it and the off-site copy does not — a MeTube video can be gone from its source site.
|
||||
**Built in the DRILL catalog 2026-10-25 night (`app-catalog-drill` `6733764`, not live):** `class: optional`, and the
|
||||
download-binds gate allows `optional` only for an app named with its ruling (`OPTIONAL_BY_RULING`); mandatory and
|
||||
unlisted stay refused. The live push is the operator's (the Monday list). **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
|
||||
media is no single app's data; whether the library gets its own household-level backup is a separate decision.
|
||||
|
||||
## 8. Removal and export assume one drive per app
|
||||
@@ -275,7 +277,9 @@ returns** (G7f).
|
||||
**[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1282]:** the „Később"
|
||||
03:30 media restart is an IN-MEMORY timer — a controller restart forgets it and only the notice stays
|
||||
(`internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`). The pick: the controller stores the due
|
||||
time and runs the restart after a restart of its own. Not built yet.
|
||||
time and runs the restart after a restart of its own. **Built 2026-10-25 night (controller `b5863a9`, unreleased):**
|
||||
`settings.media_restart_later_at`; at start `ResumeMediaLater` re-arms it, runs it at once when it passed while the
|
||||
controller was down, and clears it once it ran.
|
||||
|
||||
**[DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]:** the Sonarr/Radarr API key is READ from
|
||||
`/config/config.xml`; the „set at install" option §6 calls cleaner was not built (`internal/mediawire/mediawire.go:336-345`).
|
||||
@@ -0,0 +1,4 @@
|
||||
# R-1278 red-proof: the notice's {{if}} made false, 2026-10-25 night
|
||||
=== RUN TestR1278_TheInstallPageOffersTheKeptBackup
|
||||
r1278_kept_backup_offer_test.go:17: with a kept backup of an earlier install the page must say so and link the restore page
|
||||
--- FAIL: TestR1278_TheInstallPageOffersTheKeptBackup (0.06s)
|
||||
@@ -0,0 +1,9 @@
|
||||
# R-1280 red-proofs: the 24 h checks switched off, 2026-10-25
|
||||
verify_test.go:128: want authz: op validity window too long, got <nil>
|
||||
--- FAIL: TestVerify_RejectsPerStage (0.00s)
|
||||
--- FAIL: TestVerify_RejectsPerStage/window_longer_than_24_h (0.00s)
|
||||
FAIL
|
||||
FAIL: test_a_window_longer_than_24h_is_refused (test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused)
|
||||
AssertionError: 0 != 2 : {'authority': 'signed', 'docker_engine': '29.7.2', 'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'running'}, 'felhom-controller': {'health': 'healthy', 'id': 'aaa111', 'state': 'running'}}, 'controller': 'healthy', 'controller_docker_ok': True, 'docker_ok': True, 'network_ok': True}, 'health_before': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'running'}, 'felhom-controller': {'health': 'healthy', 'id': 'aaa111', 'state': 'running'}}, 'controller': 'healthy', 'controller_docker_ok': True, 'docker_ok': True, 'network_ok': True}, 'installed': [{'name': 'bash', 'origin': 'Debian', 'version': '5.2.37-2+b9'}, {'name': 'containerd.io', 'origin': 'Debian', 'version': '2.3.6-1~debian.13~trixie'}, {'name': 'docker-ce', 'origin': 'Debian', 'version': '5:29.8.2-1~debian.13~trixie'}, {'name': 'libc6', 'origin': 'Debian', 'version': '2.41-12+deb13u3'}, {'name': 'openssl', 'origin': 'Debian', 'version': '3.5.6-1~deb13u1'}], 'layer': 'docker', 'mode': 'apply', 'oom_check': {'detail': 'the engine reported the memory kill: OOMKilled=true and the oom event', 'exit_code': 137, 'image': 'gitea.dooplex.hu/admin/felhom-controller:0.300.0', 'oom_event': True, 'oom_killed': True, 'result': 'pass'}, 'pass_seconds': 0.0, 'pending': [{'from': '5.2.37-2+b9', 'name': 'bash', 'origin': ['Debian'], 'to': '5.2.37-2+b10'}], 'plan': {'already': 0, 'from_snapshot': 0, 'not_installed': 0, 'upgrade': 2}, 'reboot_needed': False, 'reboot_scanned': True, 'refused': None, 'release_id': 'os-docker-t1', 'repair': {'clean_after': True, 'fixed': 0, 'half_configured_before': 0, 'journal_before': 0}, 'restart_needed': [], 'seconds': 0.0, 'socket_restarted': ['felhom-controller'], 'undo': False, 'upgraded': [{'name': 'docker-ce', 'version': '5:29.8.2-1~debian.13~trixie'}, {'name': 'containerd.io', 'version': '2.3.6-1~debian.13~trixie'}], 'vmid': 9201}
|
||||
Ran 1 test in 0.013s
|
||||
FAILED (failures=1)
|
||||
@@ -0,0 +1,9 @@
|
||||
# R-1281 red-proofs, 2026-10-25 night
|
||||
## 1. the traefik-public disconnect skipped
|
||||
=== RUN TestR1281_FileBrowserLeavesTheAppNetworkAndIsServedOnTheFilesNetwork
|
||||
filebrowser_password_test.go:341: FileBrowser is still on traefik-public — every app container reaches it
|
||||
--- FAIL: TestR1281_FileBrowserLeavesTheAppNetworkAndIsServedOnTheFilesNetwork (0.00s)
|
||||
## 2. the OnlyOffice step removed
|
||||
=== RUN TestR1281_OnlyOfficeJoinsTheFilesNetworkOnlyWhileLinked
|
||||
filebrowser_password_test.go:385: a linked OnlyOffice must reach FileBrowser: map[3f2a9c@felhom-files:true felhom-controller@felhom-filebrowser:true traefik@felhom-files:true]
|
||||
--- FAIL: TestR1281_OnlyOfficeJoinsTheFilesNetworkOnlyWhileLinked (0.00s)
|
||||
@@ -0,0 +1,7 @@
|
||||
# R-1283 red-proof: the ruling's entry removed (drill catalog), 2026-10-25 night
|
||||
download-binds gate FAILED (1 of 6 download folder(s)):
|
||||
templates/metube: the download folder media/metube is declared `optional` — a nightly copy on the second drive; declare it `class: excluded` (R-1019)
|
||||
XX GENUINE (R-1283): metube's folder optional by ruling rc=1 (expected 0)
|
||||
ok FACT (R-1283): metube's folder mandatory — the ruling allows optional only rc=1 (expected 1)
|
||||
ok FACT (R-1283): metube's folder UNLISTED rc=1 (expected 1)
|
||||
FAIL: GENUINE (R-1283): metube's folder optional by ruling: rc=1 expected 0; missing ['every one']
|
||||
@@ -0,0 +1,10 @@
|
||||
# R-1285 red-proofs, 2026-10-25 night
|
||||
## agent: the policyPass call removed
|
||||
=== RUN TestR1285_AnInstalledBundleIsFollowedByThePolicyPass
|
||||
bundle_test.go:173: after the install the agent must run the policy pass (host layer, no envelope): [map[bundle:/tmp/TestR1285_AnInstalledBundleIsFollowedByThePolicyPass3671177433/001/bundle-0.143.0.json layer:host mode:bundle release_id:bundle-0.143.0 signed:map[blob_b64:eyJvcCI6ImFnZW50X2NvbmZpZ191cGRhdGUifQ== sig:SIG]]]
|
||||
--- FAIL: TestR1285_AnInstalledBundleIsFollowedByThePolicyPass (0.00s)
|
||||
## wrapper: policy_pass made to do nothing
|
||||
FAIL: test_the_policy_pass_removes_our_copy_from_a_byo_host (test_felhom_config_bundle.ByoKeepsItsSSH.test_the_policy_pass_removes_our_copy_from_a_byo_host)
|
||||
AssertionError: '/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf' unexpectedly found in {'/var/lib/felhom-install/state.json': b'{"completed": [], "mode": "byo"}', '/etc/felhom/os-trust.json': b'{
|
||||
Ran 1 test in 0.057s
|
||||
FAILED (failures=1)
|
||||
@@ -0,0 +1,4 @@
|
||||
# R-1286 red-proof: holdWhy back to the old two-way branch, 2026-10-25 night
|
||||
=== RUN TestR1286_EveryHoldReasonSaysWhatItIs
|
||||
r1286_hold_why_test.go:32: kimai (reason "unhealthy_stop"): the gate says "held after a failed restore whose rollback also failed (2026-10-14T09:06:31Z) — clear the hold to start it", want it to contain "stopped by the box for a crash loop"
|
||||
--- FAIL: TestR1286_EveryHoldReasonSaysWhatItIs (0.00s)
|
||||
@@ -0,0 +1,12 @@
|
||||
# R-1288 red-proof (retireReplacedUnit call removed), 2026-10-25
|
||||
=== RUN TestR1288_ARestoreOfASetAsideUnitRetiresTheReplacedUnitQuietly
|
||||
unit_owner_test.go:358: after the restore the primary place still holds install "bbbb000000000002"'s unit — it must be set aside at the restore
|
||||
--- FAIL: TestR1288_ARestoreOfASetAsideUnitRetiresTheReplacedUnitQuietly (0.02s)
|
||||
|
||||
# R-1282 red-proof (ResumeMediaLater a no-op)
|
||||
=== RUN TestR1282_ALaterRestartThatPassedWhileDownRunsOnceAtStart
|
||||
r1282_media_later_test.go:43: the passed restart ran 0 time(s) at start, want 1
|
||||
--- FAIL: TestR1282_ALaterRestartThatPassedWhileDownRunsOnceAtStart (2.00s)
|
||||
=== RUN TestR1282_AFutureLaterRestartIsKeptOverAStart
|
||||
r1282_media_later_test.go:74: the timer must be armed for 2026-10-25 22:29:25.783026813 +0100 CET m=+3602.004971479, got 0001-01-01 00:00:00 +0000 UTC (armed false)
|
||||
--- FAIL: TestR1282_AFutureLaterRestartIsKeptOverAStart (0.05s)
|
||||
@@ -0,0 +1,33 @@
|
||||
20:28:10 hold seen — NO switch this time
|
||||
qB prefs: up_limit 2499584 alt_up_limit 1024 mode=1
|
||||
qBittorrent: [('cc-seed-qb.bin', 'uploading', 529, 'KiB/s')] limit=1024 mode=1
|
||||
Transmission: [('cc-seed-tr.bin', 6, 1, 'KiB/s', 1.0)] {'speed-limit-up': 1, 'speed-limit-up-enabled': True}
|
||||
qBittorrent: [('cc-seed-qb.bin', 'uploading', 339, 'KiB/s')] limit=1024 mode=1
|
||||
Transmission: [('cc-seed-tr.bin', 6, 1, 'KiB/s', 1.0)] {'speed-limit-up': 1, 'speed-limit-up-enabled': True}
|
||||
== controller lines
|
||||
2026/10/25 20:28:03 [INFO] [upwatch] line test: qbittorrent held at {Bytes:1024 Alt:true Enabled:false KBps:0} (was {Bytes:3072000 Alt:true Enabled:false KBps:0})
|
||||
2026/10/25 20:28:03 [INFO] [upwatch] line test: transmission held at {Bytes:0 Alt:false Enabled:true KBps:1} (was {Bytes:0 Alt:false Enabled:false KBps:500})
|
||||
2026/10/25 20:28:03 [INFO] [upwatch] line test: 2 app(s) held — asking the agent for a test
|
||||
2026/10/25 20:29:00 [INFO] [upwatch] qbittorrent's upload limit is back as it was ({Bytes:3072000 Alt:true Enabled:false KBps:0})
|
||||
2026/10/25 20:29:00 [INFO] [upwatch] transmission's upload limit is back as it was ({Bytes:0 Alt:false Enabled:false KBps:500})
|
||||
2026/10/25 20:29:00 [INFO] [upwatch] line test on request: 94.8 Mbit/s
|
||||
2026/10/25 20:29:00 [INFO] [upwatch] the host's weekly line test: 94.8 Mbit/s (measured 2026-10-25T20:28:18Z)
|
||||
== agent lines
|
||||
Oct 25 12:41:54 pf-test felhom-agent[1117]: time=2026-10-25T20:41:54.210+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backu
|
||||
Oct 25 12:48:11 pf-test felhom-agent[11486]: time=2026-10-25T12:48:11.728-07:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a back
|
||||
Oct 25 13:13:03 pf-test felhom-agent[11486]: time=2026-10-25T13:13:03.535-07:00 level=INFO msg="local-api: line test requested" vmid=9201
|
||||
Oct 25 13:13:18 pf-test felhom-agent[11486]: time=2026-10-25T13:13:18.538-07:00 level=INFO msg="linetest: test on request refused — the uplink is not quiet" mbps=10.0
|
||||
Oct 25 13:13:18 pf-test felhom-agent[11486]: time=2026-10-25T13:13:18.539-07:00 level=INFO msg="local-api: line test request refused" vmid=9201 reason=not-quiet detail="the uplink sent 10.0 Mbit/s while held"
|
||||
Oct 25 13:28:03 pf-test felhom-agent[11486]: time=2026-10-25T13:28:03.644-07:00 level=INFO msg="local-api: line test requested" vmid=9201
|
||||
Oct 25 13:28:11 pf-test felhom-agent[11486]: time=2026-10-25T13:28:11.729-07:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=line-test
|
||||
Oct 25 13:28:18 pf-test felhom-agent[11486]: time=2026-10-25T13:28:18.647-07:00 level=INFO msg="linetest: test on request (a held torrent app, R-1250)" server=10.77.0.1:5201
|
||||
Oct 25 13:28:29 pf-test felhom-agent[11486]: time=2026-10-25T13:28:29.021-07:00 level=INFO msg="linetest: run 1: 94.8 Mbit/s" server=10.77.0.1:5201 server_bytes=119144448 server_seconds=10.058812
|
||||
Oct 25 13:28:44 pf-test felhom-agent[11486]: time=2026-10-25T13:28:44.399-07:00 level=INFO msg="linetest: run 2: 94.7 Mbit/s" server=10.77.0.1:5201 server_bytes=119013376 server_seconds=10.057222
|
||||
Oct 25 13:28:59 pf-test felhom-agent[11486]: time=2026-10-25T13:28:59.786-07:00 level=INFO msg="linetest: run 3: 94.4 Mbit/s" server=10.77.0.1:5201 server_bytes=118751232 server_seconds=10.067068
|
||||
Oct 25 13:28:59 pf-test felhom-agent[11486]: time=2026-10-25T13:28:59.786-07:00 level=INFO msg="linetest: upload line measured: 94.8 Mbit/s (best of 3)" server=10.77.0.1:5201 runs=3 duration_s=41.138682127
|
||||
== after
|
||||
qB prefs: up_limit 2499584 alt_up_limit 3072000 mode=1
|
||||
qBittorrent: [('cc-seed-qb.bin', 'uploading', 541, 'KiB/s')] limit=3072000 mode=1
|
||||
Transmission: [('cc-seed-tr.bin', 6, 360, 'KiB/s', 1.0)] {'speed-limit-up': 500, 'speed-limit-up-enabled': False}
|
||||
qBittorrent: [('cc-seed-qb.bin', 'uploading', 565, 'KiB/s')] limit=3072000 mode=1
|
||||
Transmission: [('cc-seed-tr.bin', 6, 352, 'KiB/s', 1.0)] {'speed-limit-up': 500, 'speed-limit-up-enabled': False}
|
||||
@@ -0,0 +1,27 @@
|
||||
20:13:07 hold seen — switching qBittorrent to its alternative speed
|
||||
qBittorrent: [('cc-seed-qb.bin', 'uploading', 258, 'KiB/s')] limit=3072000 mode=1
|
||||
Transmission: [('cc-seed-tr.bin', 6, 1, 'KiB/s', 1.0)] {'speed-limit-up': 1, 'speed-limit-up-enabled': True}
|
||||
qB prefs: up_limit 1024 alt_up_limit 3072000 mode=1
|
||||
qBittorrent: [('cc-seed-qb.bin', 'uploading', 210, 'KiB/s')] limit=3072000 mode=1
|
||||
Transmission: [('cc-seed-tr.bin', 6, 1, 'KiB/s', 1.0)] {'speed-limit-up': 1, 'speed-limit-up-enabled': True}
|
||||
== controller lines
|
||||
2026/10/25 19:58:03 [INFO] [upwatch] qBittorrent's upload limit is the household's — the box leaves it alone from now on
|
||||
2026/10/25 20:13:03 [INFO] [upwatch] line test: qbittorrent held at {Bytes:1024 Alt:false Enabled:false KBps:0} (was {Bytes:2499584 Alt:false Enabled:false KBps:0})
|
||||
2026/10/25 20:13:03 [INFO] [upwatch] line test: transmission held at {Bytes:0 Alt:false Enabled:true KBps:1} (was {Bytes:0 Alt:false Enabled:false KBps:500})
|
||||
2026/10/25 20:13:03 [INFO] [upwatch] line test: 2 app(s) held — asking the agent for a test
|
||||
2026/10/25 20:13:18 [INFO] [upwatch] qbittorrent's upload limit is back as it was ({Bytes:2499584 Alt:false Enabled:false KBps:0})
|
||||
2026/10/25 20:13:19 [INFO] [upwatch] transmission's upload limit is back as it was ({Bytes:0 Alt:false Enabled:false KBps:500})
|
||||
2026/10/25 20:13:19 [INFO] [upwatch] line test on request refused by the agent: not-quiet (the uplink sent 10.0 Mbit/s while held)
|
||||
== agent lines
|
||||
Oct 25 12:25:22 pf-test felhom-agent[2092]: time=2026-10-25T20:25:22.507+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backu
|
||||
Oct 25 12:41:54 pf-test felhom-agent[1117]: time=2026-10-25T20:41:54.210+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backu
|
||||
Oct 25 12:48:11 pf-test felhom-agent[11486]: time=2026-10-25T12:48:11.728-07:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a back
|
||||
Oct 25 13:13:03 pf-test felhom-agent[11486]: time=2026-10-25T13:13:03.535-07:00 level=INFO msg="local-api: line test requested" vmid=9201
|
||||
Oct 25 13:13:18 pf-test felhom-agent[11486]: time=2026-10-25T13:13:18.538-07:00 level=INFO msg="linetest: test on request refused — the uplink is not quiet" mbps=10.0
|
||||
Oct 25 13:13:18 pf-test felhom-agent[11486]: time=2026-10-25T13:13:18.539-07:00 level=INFO msg="local-api: line test request refused" vmid=9201 reason=not-quiet detail="the uplink sent 10.0 Mbit/s while held"
|
||||
== after
|
||||
qB prefs: up_limit 2499584 alt_up_limit 3072000 mode=1
|
||||
qBittorrent: [('cc-seed-qb.bin', 'uploading', 658, 'KiB/s')] limit=3072000 mode=1
|
||||
Transmission: [('cc-seed-tr.bin', 6, 208, 'KiB/s', 1.0)] {'speed-limit-up': 500, 'speed-limit-up-enabled': False}
|
||||
qBittorrent: [('cc-seed-qb.bin', 'uploading', 690, 'KiB/s')] limit=3072000 mode=1
|
||||
Transmission: [('cc-seed-tr.bin', 6, 32, 'KiB/s', 1.0)] {'speed-limit-up': 500, 'speed-limit-up-enabled': False}
|
||||
@@ -0,0 +1,13 @@
|
||||
header name read: yes
|
||||
transmission (networks: traefik-public ): no header -> 000 | forged header (household user) -> 000
|
||||
qbittorrent (networks: traefik-public ): no header -> 000 | forged header (household user) -> 000
|
||||
filebrowser networks: felhom-filebrowser felhom-files
|
||||
traefik networks: felhom-files felhom-tunnel traefik-public
|
||||
HTTP/1.1 200 OK
|
||||
1. dashboard /__files/open -> HTTP/2 302 | to files host: True
|
||||
2. files host /_felhom/sso -> HTTP/2 302 | files cookie set: True
|
||||
3. the drive's listing through traefik with the files cookie -> 200
|
||||
4. the same without any cookie (control) -> 401
|
||||
felhom-filebrowser felhom-files
|
||||
transmission (networks: traefik-public ): no header -> 000 | forged header (household user) -> 000
|
||||
qbittorrent (networks: traefik-public ): no header -> 000 | forged header (household user) -> 000
|
||||
@@ -0,0 +1,7 @@
|
||||
header name read: yes
|
||||
transmission (networks: traefik-public ): no header -> 401 | forged header (household user) -> 200
|
||||
qbittorrent (networks: traefik-public ): no header -> 401 | forged header (household user) -> 200
|
||||
filebrowser networks: felhom-filebrowser traefik-public
|
||||
traefik networks: felhom-tunnel traefik-public
|
||||
2026/10/25 22:10:12[33m GET | 401 | 172.18.0.5 | N/A | 0ms | "/api/resources?path=%2F&source=Adat"[0m
|
||||
2026/10/25 22:10:12 GET | 200 | 172.18.0.5 | felhom-hou.. | 1ms | "/api/resources?path=%2F&source=Adat"
|
||||
@@ -0,0 +1,4 @@
|
||||
candidates {'data': {'vmid': 9201, 'initialize': [{'device': '/dev/sdc', 'size_bytes': 17179869184, 'model': 'QEMU HARDDISK', 'data_bearing': False, 'mountable': False}], 'attach': []}, 'ok': True}
|
||||
init 200 {'data': {'phase': 'formatting', 'started': True}, 'ok': True}
|
||||
status {'device': '/dev/sdc', 'durable_id': '', 'error': '', 'opsign': '', 'phase': 'done', 'reason': '', 'started_at': '2026-10-25T21:57:37.534911551Z', 'updated_at': '2026-10-25T21:57:40.038690849Z', 'where': '/mnt/felhom-drives/masodik'}
|
||||
assign 200 {'data': {'assigned': '/mnt/felhom-drives/masodik', 'restart_required': True}, 'ok': True}
|
||||
@@ -0,0 +1,10 @@
|
||||
22:58:49 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH']
|
||||
22:58:49 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
22:59:44 [1] deployed, controller state=running, pinned={'metube': 'ghcr.io/alexta69/metube:2026.09.29'}
|
||||
deploy True
|
||||
drwxrwsr-x 2 1000 1000 4096 Oct 25 19:42 /mnt/felhom-drives/adat/userdata/media/metube
|
||||
67353ace24aca0fced9bb0898b2d871c4b1a799dc56bde69433b22ae8a019ac2 /mnt/felhom-drives/adat/userdata/media/metube/cc-r1283-video.mp4
|
||||
|
||||
tier2 200 {'ok': True, 'message': '2. mentés elindítva'}
|
||||
67353ace24aca0fced9bb0898b2d871c4b1a799dc56bde69433b22ae8a019ac2 /mnt/felhom-drives/masodik/backups/secondary/metube/userdata/media/metube/cc-r1283-video.mp4
|
||||
# off-site half: the rig has no off-site backup set up — not run live; governed by appbackup.tierKeeps (optional → TierSecondary only, internal/appbackup/captureset.go:191-199)
|
||||
@@ -0,0 +1,6 @@
|
||||
2026/10/25 21:58:11 [INFO] [sync] Initial sync: Sablonok frissítve — frissítve: homepage, metube, plex
|
||||
|
||||
expect n8n/docker-compose.yml ∋ N8N_DIAGNOSTICS_ENABLED=false: 1
|
||||
expect opengist/docker-compose.yml ∋ 256M: 2
|
||||
expect homepage/.felhom.yml ∋ family_gate: true: 1
|
||||
drill main 6733764, box cache 6733764: DRILL-CONFIRMED
|
||||
@@ -0,0 +1,4 @@
|
||||
mirror main 6733764
|
||||
cache HEAD cd0cf48
|
||||
https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
# R-1285 live on the rig (BYO host Tester4-84bbe7), 2026-10-25 night
|
||||
# staged: new agent 0.166.0-r1285 (main abcc1e6) installed by hand; the OLD 0.164.0 wrapper put back (no bundle-policy mode); our 0.164.0 drop-in planted (sha256 f0485da5…, sshd -T without-password)
|
||||
# then ONE signed agent_config_update, test bundle 0.166.0-r1285 (sha256 e73bdb6c…)
|
||||
Oct 25 14:53:46 pf-test felhom-os-apply[152336]: os-apply: BUNDLE START agent=0.166.0-r1285 sha=e73bdb6ce9fa6849 authority=signed files=32 write=1 same=26 kept=1 skipped=4
|
||||
Oct 25 14:53:46 pf-test felhom-os-apply[152337]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-os-apply (replaced)
|
||||
Oct 25 14:53:47 pf-test felhom-os-apply[152434]: os-apply: BUNDLE DONE agent=0.166.0-r1285 written=1 same=26 self-check=ok signers-created=False
|
||||
Oct 25 14:53:47 pf-test felhom-agent[137016]: time=2026-10-25T14:53:47.830-07:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE START agent=0.166.0-r1285 sha=e73bdb6ce9fa6849 authority=signed files=32 write=1 same=26 kept=1 skipp
|
||||
Oct 25 14:53:47 pf-test felhom-agent[137016]: time=2026-10-25T14:53:47.830-07:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE WROTE /usr/local/sbin/felhom-os-apply (replaced)"
|
||||
Oct 25 14:53:47 pf-test felhom-agent[137016]: time=2026-10-25T14:53:47.830-07:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.166.0-r1285 written=1 same=26 self-check=ok signers-created=False"
|
||||
Oct 25 14:53:47 pf-test felhom-agent[137016]: time=2026-10-25T14:53:47.830-07:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.166.0-r1285 bundle="{\"agent_version\": \"0.166.0-r1285\", \"authorit
|
||||
Oct 25 14:53:47 pf-test felhom-os-apply[152438]: os-apply: BUNDLE POLICY REMOVED /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf (ours, on a BYO host — D43, R-1285)
|
||||
Oct 25 14:53:48 pf-test felhom-os-apply[152447]: os-apply: BUNDLE POLICY DONE install_mode=byo removed=1
|
||||
Oct 25 14:53:48 pf-test felhom-agent[137016]: time=2026-10-25T14:53:48.034-07:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE POLICY REMOVED /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf (ours, on a BYO host — D43, R-12
|
||||
Oct 25 14:53:48 pf-test felhom-agent[137016]: time=2026-10-25T14:53:48.034-07:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE POLICY DONE install_mode=byo removed=1"
|
||||
Oct 25 14:53:48 pf-test felhom-agent[137016]: time=2026-10-25T14:53:48.034-07:00 level=INFO msg="osupdate: bundle policy pass done" op=agent_config_update agent_version=0.166.0-r1285 bundle_policy="{\"install_mode\": \"byo\", \"removed\": [
|
||||
Oct 25 14:53:48 pf-test felhom-agent[137016]: time=2026-10-25T14:53:48.913-07:00 level=WARN msg="signedjobs: signed op COMPLETED" job=14de31ae946b9f76 op=agent_config_update
|
||||
total 8
|
||||
drwxr-xr-x 2 root root 4096 Oct 25 14:53 .
|
||||
drwxr-xr-x 4 root root 4096 Oct 25 12:21 ..
|
||||
permitrootlogin yes
|
||||
5
|
||||
end-of-wait
|
||||
|
||||
[exited with code 0]
|
||||
@@ -0,0 +1,22 @@
|
||||
20:40:45Z [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
20:43:06Z [1] deployed, controller state=running, pinned={'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:17-alpine'}
|
||||
20:43:06Z gate: recipes is gated — passed as the household (cookie set)
|
||||
20:43:19Z tandoor: createsuperuser :: Running django-vite in production mode (no HMR) Superuser created successfully.
|
||||
20:43:24Z tandoor: seeded superuser drill7c6d3e
|
||||
20:43:24Z [4] Mentés most -> 200 {'ok': True, 'message': 'Mentés elindítva'}
|
||||
20:44:22Z [5] stop 200, remove KEEPING data -> 200 {'ok': True, 'data': {'removed': 'tandoor', 'volumes_removed': ['tandoor_tandoor_media', 'tandoor_tandoor_postgres_data', 'tandoor_tandoor_static'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [],
|
||||
20:44:27Z [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
20:46:33Z [1] deployed, controller state=running, pinned={'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:17-alpine'}
|
||||
20:49:57Z set-aside dirs after the reinstall's backup: 0
|
||||
20:49:58Z POST /backup/kept/restore -> HTTP/2 302
|
||||
20:51:38Z restore: {"running": false, "op": "restore", "stack": "tandoor", "started_at": "2026-10-25T20:49:58.398659098Z", "last": {"op": "restore", "stack": "tandoor", "ok": true, "message": "A(z) tandoor: 3 adatkötet
|
||||
20:52:08Z tandoor: readback of the seeded account found=True
|
||||
20:52:08Z item back: True
|
||||
20:52:08Z controller lines since the restore:
|
||||
2026/10/25 20:51:33 [INFO] [backup] tandoor: the unit of the install the restore replaced (7dc433bf4851beb7) is set aside whole → /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T205133Z — replaced by the restored install e30f6554375a455b; no
|
||||
|
||||
20:52:14Z set-aside dirs after the restore: 0
|
||||
20:52:14Z [4] Mentés most -> 200 {'ok': True, 'message': 'Mentés elindítva'}
|
||||
20:52:47Z after the next backup — set-aside lines: 0 | set-aside dirs: 0
|
||||
20:53:28Z [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'tandoor', 'volumes_removed': ['tandoor_tandoor_media', 'tandoor_tandoor_postgres_data', 'tandoor_tandoor_static'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [],
|
||||
20:53:36Z after remove: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '', 'kept': '', 'subdomain': '404'}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,9 @@
|
||||
9201
|
||||
pf-test
|
||||
- the escrow blob in the hub, if one exists (operator UI)
|
||||
|
||||
[0;32m[OK][0m UNINSTALL complete — removed: guest 9201, the felhom-agent (unit/sudoers/binary/state/config+baks/user + selfupdate-artifacts/shared-parent/mkfs-wrapper/pbs-apply-wrapper/hook-snippet/dnsmasq-snippets), the pveum role/user/token/ACL, the felhom pool, and /var/lib/felhom-install/state.json.
|
||||
[0;32m[INFO][0m golden vzdump: left in place (--remove-golden to remove).
|
||||
[0;32m[INFO][0m NOTE: the 'sudo' package was left INSTALLED (a system package, never ours to remove). dnsmasq is removed only when the install-time record says Felhom installed it; a resolver that pre-dated Felhom is never touched. The host record still exists in the hub — remove it there if desired.
|
||||
UNINSTALL_RC=0
|
||||
Connection to 192.168.0.150 closed.
|
||||
@@ -0,0 +1,33 @@
|
||||
#!/usr/bin/env python3
|
||||
"""r1281household.py — R-1281's positive control on the night's rig: the household still reaches its files through
|
||||
traefik after FileBrowser left traefik-public. The walk the dashboard's „Fájlok ↗" makes: GET /__files/open on the
|
||||
dashboard host (with the household's session) → a one-time pass → GET /_felhom/sso on files.<domain> → the files cookie
|
||||
→ the drive's listing through traefik (forwardAuth adds the secret header). No secret is printed."""
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import rigw # noqa: E402
|
||||
|
||||
w = rigw.w
|
||||
w.login()
|
||||
sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip()
|
||||
files_host = "files." + w.DOMAIN
|
||||
r = w.sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}",
|
||||
f"{w.BASE}/__files/open?to=%2F"])
|
||||
loc = w._loc(r.stdout)
|
||||
print("1. dashboard /__files/open ->", (r.stdout or "").split("\n")[0].strip(), "| to files host:", files_host in loc)
|
||||
m = re.search(r"https?://[^/]+(/_felhom/sso\?[^\s]+)", loc)
|
||||
if not m:
|
||||
sys.exit("no pass handed out")
|
||||
r = w.sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", f"Host: {files_host}", f"{w.BASE}{m.group(1)}"])
|
||||
ck = re.search(r"(?im)^set-cookie:\s*(felhom_files=[^;\r\n]+)", r.stdout or "")
|
||||
print("2. files host /_felhom/sso ->", (r.stdout or "").split("\n")[0].strip(), "| files cookie set:", bool(ck))
|
||||
if not ck:
|
||||
sys.exit("no files cookie")
|
||||
url = f"{w.BASE}/api/resources?path=%2F&" + "sou" + "rce=Adat"
|
||||
r = w.sh(["curl", "-sk", "-o", "/dev/null", "-w", "%{http_code}", "-H", f"Host: {files_host}", "-H", f"Cookie: {ck.group(1)}", url])
|
||||
print("3. the drive's listing through traefik with the files cookie ->", r.stdout)
|
||||
r = w.sh(["curl", "-sk", "-o", "/dev/null", "-w", "%{http_code}", "-H", f"Host: {files_host}", url])
|
||||
print("4. the same without any cookie (control) ->", r.stdout)
|
||||
@@ -0,0 +1,58 @@
|
||||
#!/usr/bin/env python3
|
||||
"""r1288rig.py — R-1288 live on the night's rig (rc.2): the Tester 1 order with a throwaway Tandoor — install, back up,
|
||||
remove keeping backups, reinstall (its first backup sets install A aside: one alarm), restore A's set-aside copy →
|
||||
the replaced install's unit is set aside AT the restore („replaced by the restored install"), no second alarm; the
|
||||
next backup sets nothing aside. Removed through the product at the end."""
|
||||
import json, os, re, sys, time
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import rigw # noqa: E402 (must come first: it points the walk at the rig)
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools"))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../restore-truth-2026-10-25/tools"))
|
||||
import sweep as sw # noqa: E402
|
||||
import unitprobe2 as up # noqa: E402
|
||||
|
||||
w, fxm = sw.w, sw.fxm
|
||||
APP = "tandoor"
|
||||
AS = "/mnt/felhom-drives/adat/felhom-data/backups/set-aside/tandoor"
|
||||
|
||||
|
||||
def say(*a):
|
||||
print(time.strftime("%H:%M:%SZ", time.gmtime()), *a, flush=True)
|
||||
|
||||
|
||||
w.say = say
|
||||
fx = fxm.FIXTURES[APP]
|
||||
sub = getattr(fx, "sub", APP)
|
||||
w.login()
|
||||
t0 = int(time.time())
|
||||
try:
|
||||
w.deploy(APP, sub) and w.wait_app(sub, "/", tries=90, delay=5)
|
||||
tok = fx.seed(w, sub, say)
|
||||
sw.backup_now({}, say)
|
||||
sw.remove(APP, keep=True, say=say)
|
||||
w.deploy(APP, sub, dict(w.GENERATED.get(APP) or {}))
|
||||
for _ in range(60):
|
||||
time.sleep(10)
|
||||
if sw.g(f"docker logs --since {t0} felhom-controller 2>&1 | grep -c 'recovery unit of an earlier install'").strip() not in ("", "0"):
|
||||
break
|
||||
say("set-aside dirs after the reinstall's backup:", sw.g(f"find /mnt/felhom-drives -path '*set-aside/tandoor/*' -maxdepth 6 -mindepth 6 -type d 2>/dev/null | wc -l").strip())
|
||||
html = up.page("/backups/restore")
|
||||
m = re.search(r'action="/backup/kept/restore".*?name="id" value="([0-9a-f]{16})"', html, re.S)
|
||||
t_r = int(time.time())
|
||||
head, loc = up.post_form("/backup/kept/restore", {"id": m.group(1), "confirm": "1"})
|
||||
say("POST /backup/kept/restore ->", head)
|
||||
rec = {}
|
||||
up.wait_restore("kept", rec)
|
||||
say("restore:", rec["kept"]["status"][:200])
|
||||
time.sleep(20)
|
||||
say("item back:", bool(tok) and bool(fx.verify(w, sub, tok, say)))
|
||||
say("controller lines since the restore:")
|
||||
print(sw.g(f"docker logs --since {t_r} felhom-controller 2>&1 | grep -E 'replaced by the restored install|SET ASIDE|set aside|Event pushed: backup_run_failures' | cut -c1-260"))
|
||||
say("set-aside dirs after the restore:", sw.g(f"find /mnt/felhom-drives -path '*set-aside/tandoor/*' -maxdepth 6 -mindepth 6 -type d 2>/dev/null | wc -l").strip())
|
||||
t_b = int(time.time())
|
||||
sw.backup_now({}, say)
|
||||
say("after the next backup — set-aside lines:", sw.g(f"docker logs --since {t_b} felhom-controller 2>&1 | grep -c 'recovery unit of an earlier install'").strip(),
|
||||
"| set-aside dirs:", sw.g(f"find /mnt/felhom-drives -path '*set-aside/tandoor/*' -maxdepth 6 -mindepth 6 -type d 2>/dev/null | wc -l").strip())
|
||||
finally:
|
||||
sw.remove(APP, keep=False, say=say)
|
||||
say("after remove:", sw.leftovers(APP, sub))
|
||||
@@ -124,7 +124,7 @@ stopping line that lies.
|
||||
| **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC |
|
||||
| **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator |
|
||||
|
||||
## Apps & catalog — 16 rows (P3 6, P4 10)
|
||||
## Apps & catalog — 17 rows (P3 6, P4 11)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -145,6 +145,7 @@ stopping line that lies.
|
||||
| **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | WATCHING — **2026-10-22 night:** not reproducible tonight; the Immich server log of 19:38 is gone (Immich restarted at 02:16 for the off-site dump). What the household sees: the upload fails after 10 minutes and the Immich phone app sends it again. | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC |
|
||||
| **R-1267** | Apps & catalog | P4 | **Every recreate of Immich leaves one empty anonymous Docker volume behind: the image declares `VOLUME /data`, the template does not map it.** Found 2026-10-24 (review round 2, while putting 9202 back): 9202 holds 296 empty anonymous volumes; each night chain's recreate of `immich-server` made a new one (12:58, 13:00, 13:01 — the three test runs; the two dangling ones this session made were removed by name, nothing pruned). The photos are NOT affected: Immich picks `/usr/src/app/upload` whenever both folders exist (`detectMediaLocation` in the image's `server/dist/services/storage.service.js:28-39`), and the template binds that folder (`templates/immich/docker-compose.yml:49`) — NOT-A-FINDING for data placement. **Seen again 2026-10-24 night on Tester 1:** privatebin's image declares `/run`, `/tmp` and `/var/lib/nginx/tmp` as volumes — each recreate (the night dump's stop/start) leaves three empty anonymous volumes; the same class as Immich's `/data` (`audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt`). | OPEN **2026-10-24 (catalog sweep):** the same class for OnlyOffice and RomM (RomM's image volume `/romm` holds only the empty mount points of its binds); after every catalog app had been removed WITH data, the sweep box still held 52 such volumes (PrivateBin 21, OnlyOffice 8, RomM 8, Immich 3, and 12 from 12:50–12:52 before the batch, holding the same nginx folders as PrivateBin's) — remove-with-data does not see them (no compose project label). All small (4–72 KB), no household data (`audits/catalog-sweep-2026-10-24/_box/anon-volumes.txt`). | — | declare the image's VOLUME paths as tmpfs or named volumes in the catalog (Immich, privatebin), or remove the dangling ones a recreate leaves, by name | CC |
|
||||
| **R-1268** | Backup & restore | P3 | **Remove with „remove backups” leaves a restore's undo copy on the drive (`backups/restore-undo/<app>/`).** Seen 2026-10-24 night on Tester 1 (restore-holes Part 1, run 1): MeTube was held for support after a failed crash put-back, then removed with `remove_backups=true` — the 1.5 GB copy stayed (the removal deletes `backups/primary` and the second-drive mirror, not `restore-undo`), and nothing lists it (`audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt`; removed by hand, CC's own test data). The copy is kept on purpose while a hold exists; after the app is gone it is orphaned. | OPEN | — | Remove with `remove_backups` also deletes the app's `restore-undo` folder (under the same path guard), or lists it as a kept copy | CC |
|
||||
| **R-1292** | Apps & catalog | P4 | **qBittorrent's 50 % rule reads and writes the limit of the CURRENT speed mode: with the household's alternative speed on, it sets the alternative limit, and a later switch reads as „the household changed it" — the rule stops for good.** Measured 2026-10-25 night on qBittorrent 5.2.4 (`transfer/uploadLimit`/`setUploadLimit` follow the mode; `audits/night-2026-10-25/r1261-qbittorrent-api-measured.txt`); found while fixing R-1261 (the hold now names its slot). | OPEN | — | make the rule own the NORMAL slot only (`up_limit` through `app/preferences`, as the hold does) + a test with the alternative mode on (red first); one controller release | CC |
|
||||
|
||||
|
||||
## App updates — 5 rows (P2 1, P3 4)
|
||||
@@ -176,7 +177,7 @@ stopping line that lies.
|
||||
| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. | OPEN | — | one attended pass over the restore paths with these four as tests | CC |
|
||||
| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
|
||||
| **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
|
||||
| **R-1278** | Backup & restore | P3 | **A volume app's reinstall does not offer its kept backup.** „Remove — keep my data” removes a volume app's named volumes and keeps only its backup; the kept-data question is about drive folders only (`07` §6.5, decision 36), so the reinstall starts empty and says nothing. Since D47 (2026-10-25) the earlier install's unit is set aside and listed on the restore page („Megőrzött másolatok”, „Visszaállítás”) — but the install page does not point to it (`audits/restore-truth-2026-10-25/README.md`). | OPEN | — | Design: at install, when a set-aside or removed-app unit of the app exists, say so and offer it beside a fresh start; a test | CC |
|
||||
| **R-1278** | Backup & restore | P3 | **A volume app's reinstall does not offer its kept backup.** „Remove — keep my data” removes a volume app's named volumes and keeps only its backup; the kept-data question is about drive folders only (`07` §6.5, decision 36), so the reinstall starts empty and says nothing. Since D47 (2026-10-25) the earlier install's unit is set aside and listed on the restore page („Megőrzött másolatok”, „Visszaállítás”) — but the install page does not point to it (`audits/restore-truth-2026-10-25/README.md`). | **VERIFY** — fixed on main 2026-10-25 night (controller `fbec5cb`), waits for the controller release: for an app not installed, when the box keeps a backup of an earlier install (set-aside or removed-app unit), the install page says the install starts empty and links the restore page (HU and EN; render test per branch, red-proved, `redproof-r1278.txt`). Not seen live. | — | Design: at install, when a set-aside or removed-app unit of the app exists, say so and offer it beside a fresh start; a test | CC |
|
||||
| **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
|
||||
| **R-164** | Backup & restore | P4 | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
|
||||
| **R-213** | Backup & restore | P4 | **Putting files back in place — the half the recovery screen deliberately does not do.** The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: *a screen that unlocks and then offers to overwrite is two decisions wearing one button*. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. **The operator named its requirement: a live-versus-backup comparison** — the customer must be able to see what would change before anything is overwritten | **DEFERRED** — operator ruling 2026-10-12 (S18, `09` §3 decision 213): designed after the first customers. Stays P4; re-dated: look again when the first paying household has run 30 days. **OPEN — not started, deliberately** | the comparison design (nothing exists for it yet) | After the first customers | Operator + CC |
|
||||
@@ -212,9 +213,9 @@ stopping line that lies.
|
||||
| **R-1232** | Backup & restore | P2 | **A reinstalled box may take its first whole-box copy to ep0 up to a week late: the old box's copy in the same household space seems to count as its last one.** Test night 2026-10-23 (`audits/night-2026-10-23/P4-testnight/`): the new Tester4 box (guest 9201) had its PBS tier applied at 19:28 UTC, but no whole-box run to ep0 came all night; the only archive in the household's ep0 space is the rehearsal box's `ct/9201` of 2026-10-20 (same guest number), and the tier's cadence is 168 h. The agent's due-check reads the newest archive for that vmid on the tier's storage (`localapi/server.go` `newestArchiveOn`) — so the old box's copy reads as „2.6 days old, not due”. If so, the ep0 copy is the OLD box's for up to a week. PLAUSIBLE — read from the code and the night's silence, not proven by a log line (the controller logs the due-check only at debug). | OPEN | — | Prove: read `GET /backup/tiers` for the box (the agent's answer per tier) on the next reinstall; fix: a tier's „last backup” counts only archives this host wrote (or after the host's enrollment); an agent test with a foreign archive in the namespace | CC |
|
||||
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
|
||||
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
|
||||
| **R-1283** | Backup & restore | P3 | **MeTube's downloads are excluded from every backup — make them `optional` (D44 #25).** Catalog `c50adfc` (R-1019) marks the download folders of qBittorrent, Transmission, NZBGet, SABnzbd and MeTube `excluded` (`app-catalog-felhom.eu/templates/metube/.felhom.yml:33`). Risk: a MeTube video may be gone from its source site, so after a drive failure it cannot be fetched again. Pick: MeTube's `media/metube` becomes `optional` — the second-drive copy carries it, the off-site copy does not; NZBGet and SABnzbd stay excluded (re-downloadable). Cost: second-drive space for MeTube's folder. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §7. | OPEN | — | change the class in the catalog template, run the catalog gates (the download-binds gate and its decoys must accept MeTube as `optional`), check on a Tier-0 box that a second-drive copy carries the folder and the off-site run does not | CC |
|
||||
| **R-1286** | Backup & restore | P4 | **demo-hp's kimai has been HELD since 2026-10-14 09:06Z („held after a failed restore whose rollback also failed") and no row names it.** Seen 2026-10-25 by the D47 upgrade count (attended release run): kimai is the one app of 12 on demo-hp without an install id, because the start-up step skips a held app; `app.yaml` says deployed, no container exists, boot reconciliation leaves it held, the night update leg has skipped it since at least 2026-10-16 (`audits/burndown-apps-2026-10-16/P1/night-chain-log.txt`). A test box; the hold is the designed fail-closed state (R-379), but 11 days with no record is how a held household app would go unseen too. | OPEN | — | read why the 2026-10-14 restore and its rollback failed (demo-hp's controller log of that day, if kept; the hold record); then clear the hold (`--clear-restore-hold kimai` + a restart) or remove the app through the product; check that a held app raises a standing operator signal, else file that | CC |
|
||||
| **R-1288** | Backup & restore | P4 | **After the household restores an earlier install's set-aside copy, the next backup sets the REINSTALL's own unit aside too — a second kept copy on the restore page and a second operator alarm.** Seen 2026-10-25 on Tester 1 (0.343.0, Tandoor, attended release run): reinstall → its first backup set install A's unit aside (17:31:55) and wrote install B's unit; restoring A's copy makes the app install A again („the restored data belongs to install 795457b5… — its backups follow it (D47)"); the next 20-minute backup step (17:36:54) found B's unit in `primary/` and set it aside. Nothing is lost; the second copy is the fresh reinstall's near-empty data. D47's design does what it says; the question is whether a restore of a set-aside unit should retire the unit it replaces instead of leaving it for the next writer. | OPEN | — | read `07` §8.2d/§8.2e for the intended end state; if the restore should retire B's unit: move it aside AT the restore (one line, no alarm, „replaced by the restored install") with a test; else document the second copy and word its alarm (R-1287) | CC |
|
||||
| **R-1283** | Backup & restore | P3 | **MeTube's downloads are excluded from every backup — make them `optional` (D44 #25).** Catalog `c50adfc` (R-1019) marks the download folders of qBittorrent, Transmission, NZBGet, SABnzbd and MeTube `excluded` (`app-catalog-felhom.eu/templates/metube/.felhom.yml:33`). Risk: a MeTube video may be gone from its source site, so after a drive failure it cannot be fetched again. Pick: MeTube's `media/metube` becomes `optional` — the second-drive copy carries it, the off-site copy does not; NZBGet and SABnzbd stay excluded (re-downloadable). Cost: second-drive space for MeTube's folder. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §7. | **VERIFY** — built in the DRILL catalog 2026-10-25 night (`app-catalog-drill` `6733764`), waits for the operator's catalog push: MeTube `optional`; the download-binds gate allows it by a named ruling only (red-proved, `redproof-r1283.txt`). **Live on the rig (drill `6733764`):** a 3 MB file in MeTube's folder → „Tier 2 copied metube … 1 leg(s)", the same sha256 on the second drive (`rig/r1283-live.txt`). The off-site half NOT live (the rig has no off-site): it is `appbackup.tierKeeps` (optional → the second drive only). **Earlier state:** OPEN | — | change the class in the catalog template, run the catalog gates (the download-binds gate and its decoys must accept MeTube as `optional`), check on a Tier-0 box that a second-drive copy carries the folder and the off-site run does not | CC |
|
||||
| **R-1286** | Backup & restore | P4 | **demo-hp's kimai has been HELD since 2026-10-14 09:06Z („held after a failed restore whose rollback also failed") and no row names it.** Seen 2026-10-25 by the D47 upgrade count (attended release run): kimai is the one app of 12 on demo-hp without an install id, because the start-up step skips a held app; `app.yaml` says deployed, no container exists, boot reconciliation leaves it held, the night update leg has skipped it since at least 2026-10-16 (`audits/burndown-apps-2026-10-16/P1/night-chain-log.txt`). A test box; the hold is the designed fail-closed state (R-379), but 11 days with no record is how a held household app would go unseen too. | **WAITING-ON-OPERATOR** — read 2026-10-25 night: the hold is NOT a failed restore — `reason: unhealthy_stop`, `crash_loop`, 2026-10-14 09:06:31Z, during the night power drill on demo-hp (8 real cuts); every surface misnamed it — fixed on main (controller `838c834`, `holdWhy`, red-proved). A held app gives the operator no lasting signal → R-1291. **The pick (STATUS):** press Start (the household's Start lifts this hold; a new crash loop holds it again) or remove Kimai through the product. **Earlier state:** OPEN | — | read why the 2026-10-14 restore and its rollback failed (demo-hp's controller log of that day, if kept; the hold record); then clear the hold (`--clear-restore-hold kimai` + a restart) or remove the app through the product; check that a held app raises a standing operator signal, else file that | CC |
|
||||
| **R-1288** | Backup & restore | P4 | **After the household restores an earlier install's set-aside copy, the next backup sets the REINSTALL's own unit aside too — a second kept copy on the restore page and a second operator alarm.** Seen 2026-10-25 on Tester 1 (0.343.0, Tandoor, attended release run): reinstall → its first backup set install A's unit aside (17:31:55) and wrote install B's unit; restoring A's copy makes the app install A again („the restored data belongs to install 795457b5… — its backups follow it (D47)"); the next 20-minute backup step (17:36:54) found B's unit in `primary/` and set it aside. Nothing is lost; the second copy is the fresh reinstall's near-empty data. D47's design does what it says; the question is whether a restore of a set-aside unit should retire the unit it replaces instead of leaving it for the next writer. | **VERIFY** — fixed on main 2026-10-25 night (controller `b5863a9`; the brief's safest reading, `07` §8.2d), waits for the controller release: the replaced install's unit set aside AT the restore, no alarm. **Live on the rig (rc.2):** the line, the account back, the next backup set nothing aside, one set-aside mail only (`rig/r1288-live.txt`). **Earlier state:** OPEN | — | read `07` §8.2d/§8.2e for the intended end state; if the restore should retire B's unit: move it aside AT the restore (one line, no alarm, „replaced by the restored install") with a test; else document the second copy and word its alarm (R-1287) | CC |
|
||||
|
||||
## Storage & devices — 14 rows (P3 5, P4 9)
|
||||
|
||||
@@ -233,7 +234,7 @@ stopping line that lies.
|
||||
| **R-1213** | Storage & devices | P4 | **Operator detail still on two household pages:** the registered drive card's `ext4 · /dev/sdb1 · Model` line (`templates/storage.html`; W14 hid only the agent drive cards), and the heading „Távoli mentés (3. mentés) — titkosított, offsite" with the English „offsite". Found by the night helper 2026-10-21 (W11/W14 follow-up). | OPEN | — | Move the line behind „Részletek"; „…titkosított, házon kívüli" | CC |
|
||||
| **R-1275** | Storage & devices | P4 | **An English household sees the drive names in Hungarian** on the install page (and wherever a storage label shows): „Belső tárhely — 52.0 GB free (default)”, „Külső tárhely (hdd_1)” (catalog sweep 2026-10-24, `audits/catalog-sweep-2026-10-24/navidrome/page-install-en.txt`). The label is stored once, in Hungarian, when the drive is found (`felhom-controller/controller/internal/settings/settings.go` `InferStorageLabel`); a household may rename it. | OPEN | — | Render a label that still equals the inferred default in the reader's language; a test per language | CC |
|
||||
| **R-1276** | Storage & devices | P4 | **„Remove with data” leaves the app's empty parent folder** (`<drive>/appdata/<app>`) when the template binds a subfolder: Grimmory, RomM, Paperless-ngx (catalog sweep 2026-10-24; 4 KB each; the bound folders were removed). Harmless (empty, not counted as old data), untidy for a household that browses the drive. | OPEN | — | Remove the parent when it is empty after the bound folders go; a test | CC |
|
||||
| **R-1282** | Storage & devices | P3 | **The „Később" 03:30 media restart is forgotten by a controller restart — store the due time (D44 #24).** The restart is an in-memory timer (`felhom-controller/controller/internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`); after a controller restart only the notice stays. Risk: media apps do not see the new drives until the household presses „now". Pick: store the due time and run the restart after a controller restart (a due time already past runs at once, within the same rules). Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §11. | OPEN | — | a small controller change + a test that a stored due time survives a restart and fires once; rides a controller release | CC |
|
||||
| **R-1282** | Storage & devices | P3 | **The „Később" 03:30 media restart is forgotten by a controller restart — store the due time (D44 #24).** The restart is an in-memory timer (`felhom-controller/controller/internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`); after a controller restart only the notice stays. Risk: media apps do not see the new drives until the household presses „now". Pick: store the due time and run the restart after a controller restart (a due time already past runs at once, within the same rules). Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §11. | **VERIFY** — fixed on main 2026-10-25 night (controller `b5863a9`), waits for the controller release: the „later" time stored, resumed at start, once (`r1282_media_later_test.go`, red-proved). **Earlier state:** OPEN | — | a small controller change + a test that a stored due time survives a restart and fires once; rides a controller release | CC |
|
||||
|
||||
## Security & access — 26 rows (P1 2, P2 4, P3 14, P4 6)
|
||||
|
||||
@@ -263,8 +264,8 @@ stopping line that lies.
|
||||
| **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f <conf>` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC |
|
||||
| **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC |
|
||||
| **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator |
|
||||
| **R-1280** | Security & access | P3 | **A signed operation's window has no maximum length — cap it at 24 h (D44 #22).** The agent (`felhom-agent/internal/authz/verifier.go`) and the root wrapper (`configs/felhom-os-apply`) check `now ∈ [issued_at, expires_at]` with a 2-minute skew on `issued_at`, but neither checks `expires_at − issued_at`; `felhom-opsign` defaults to 30 min (`cmd/felhom-opsign/main.go:57`) and the nonce store keeps a record until its expiry. Risk: a signed job made with a long window by mistake stays usable for that whole time. Pick: the agent and the root wrapper refuse a window longer than 24 h. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/04-control-plane-authorization.md` §2. | OPEN | — | one agent change in both checkers + a test that a 24 h + 1 s window is refused (red first) and a 24 h one is accepted; rides an agent release (agent releases stop for the operator's vouch) | CC |
|
||||
| **R-1281** | Security & access | P2 | **Every app container can reach FileBrowser directly; the only guard is the secret header name — take FileBrowser off the app networks (D44 #23).** FileBrowser trusts its proxy header (`auth.methods.proxy`); every app container can reach `filebrowser:80` without traefik; the guard is that the header NAME is a per-box secret that traefik's forwardAuth strips from client requests (`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`). Risk: an app that learns the name (a hacked app on the box) reads and writes every household file. Pick: FileBrowser joins only the network traefik uses, so no app container reaches it directly. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/13-threat-model.md` §5. | OPEN | — | a compose/network change in the controller's infra render + a test on the rendered networks; a live check on a Tier-0 box that a forged header from an app container no longer reaches FileBrowser (the 2026-10-18 measurement, now refused) while the dashboard's file manager still signs in; one controller release | CC |
|
||||
| **R-1280** | Security & access | P3 | **A signed operation's window has no maximum length — cap it at 24 h (D44 #22).** The agent (`felhom-agent/internal/authz/verifier.go`) and the root wrapper (`configs/felhom-os-apply`) check `now ∈ [issued_at, expires_at]` with a 2-minute skew on `issued_at`, but neither checks `expires_at − issued_at`; `felhom-opsign` defaults to 30 min (`cmd/felhom-opsign/main.go:57`) and the nonce store keeps a record until its expiry. Risk: a signed job made with a long window by mistake stays usable for that whole time. Pick: the agent and the root wrapper refuse a window longer than 24 h. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/04-control-plane-authorization.md` §2. | **VERIFY** — fixed on main 2026-10-25 night (agent `d912307`), waits for the agent release + the operator's vouch: 24 h in the verifier, the root wrapper and felhom-opsign (red-proved, `redproof-r1280.txt`). **Earlier state:** OPEN | — | one agent change in both checkers + a test that a 24 h + 1 s window is refused (red first) and a 24 h one is accepted; rides an agent release (agent releases stop for the operator's vouch) | CC |
|
||||
| **R-1281** | Security & access | P2 | **Every app container can reach FileBrowser directly; the only guard is the secret header name — take FileBrowser off the app networks (D44 #23).** FileBrowser trusts its proxy header (`auth.methods.proxy`); every app container can reach `filebrowser:80` without traefik; the guard is that the header NAME is a per-box secret that traefik's forwardAuth strips from client requests (`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`). Risk: an app that learns the name (a hacked app on the box) reads and writes every household file. Pick: FileBrowser joins only the network traefik uses, so no app container reaches it directly. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/13-threat-model.md` §5. | **VERIFY** — fixed on main 2026-10-25 night (controller `7ea942a`), waits for the controller release: FileBrowser served on `felhom-files` (traefik + OnlyOffice while linked), off traefik-public. **Live on the rig (rc.3):** before, both app containers signed in with the forged header (200); after, no answer (`000`); the household's „Fájlok ↗" walk 200, no cookie 401 (`rig/r1281-*.txt`). The migration live too: FileBrowser put back on traefik-public by hand → the next 5-minute round logged „left traefik-public — no app container reaches it any more" and the apps got no answer again. OnlyOffice's join by test only (not installed on the rig). On the release every box re-creates FileBrowser once (its compose comment changed). **Earlier state:** OPEN | — | a compose/network change in the controller's infra render + a test on the rendered networks; a live check on a Tier-0 box that a forged header from an app container no longer reaches FileBrowser (the 2026-10-18 measurement, now refused) while the dashboard's file manager still signs in; one controller release | CC |
|
||||
|
||||
## Box system & updates — 13 rows (P2 1, P3 7, P4 5)
|
||||
|
||||
@@ -282,14 +283,14 @@ stopping line that lies.
|
||||
| **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
|
||||
| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
|
||||
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
|
||||
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
|
||||
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
|
||||
|
||||
|
||||
## Monitoring & notifications — 23 rows (P2 1, P3 16, P4 6)
|
||||
## Monitoring & notifications — 24 rows (P2 1, P3 17, P4 6)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | OPEN — **now in controller 0.343.0 (released 2026-10-25); the three edges act only on a box whose agent is ≥ 0.165.0 (it alone reports `waiting_for_quiet`) — fix before any household gets agent 0.165.0 by signed update** | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC |
|
||||
| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | **VERIFY** — fixed on main 2026-10-25 night (controller `a79a292`), waits for the controller release (before any household gets agent 0.165.0 by signed update): every running seeder held, a removed app's record dropped, qBittorrent's slot named (`up_limit`/`alt_up_limit`). Red-proofs `audits/night-2026-10-25/redproof-r1261.txt`. **Live on the night's rig** (BYO VM, agent 0.165.0, rc.1, two seeders): both held, 94.8 Mbit/s, both limits back; a speed-mode switch mid-test left no 1 KiB/s (`rig/hold-watch*.txt`); the removed-app case by test only. **Earlier state:** OPEN — **now in controller 0.343.0 (released 2026-10-25); the three edges act only on a box whose agent is ≥ 0.165.0 (it alone reports `waiting_for_quiet`) — fix before any household gets agent 0.165.0 by signed update** | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC |
|
||||
| **R-1262** | Monitoring & notifications | P3 | **The upload watch treats ANY viewer as an explanation for any amount of upload, so a hacked Plex can send freely while one family member watches.** Found 2026-10-24 (SEC-1 X1): `felhom-controller/controller/internal/upwatch/watch.go:275,299`; `08` §9 states the rule, its limits list did not name this case. | OPEN | — | propose a cap (viewers × a bitrate ceiling) for the operator's R-1244 design; `08`'s limits list names the case meanwhile | CC |
|
||||
| **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
|
||||
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
|
||||
@@ -312,6 +313,7 @@ stopping line that lies.
|
||||
| **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
|
||||
| **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
|
||||
| **R-1287** | Monitoring & notifications | P3 | **A D47 set-aside mails the operator as „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" — no backup failed; the body says the right thing („its backup belonged to an earlier install … SET ASIDE whole").** Seen 2026-10-25 on Tester 1 with controller 0.343.0 (attended release run): two such mails at 17:31:56Z and 17:36:55Z. The controller sends the set-aside as `backup_run_failures` leg `unit-set-aside`, `Failed: 1` (so the hub never drops a line that claims zero); the hub's subject for that event is fixed („N of M apps failed to back up", `hub/internal/notify/templates.go:353`). The CLAUDE.md corollary: an alarm's text must say what it counts. After the floor, every household reinstall mails the operator a red „failed to back up". | **VERIFY** — fixed on main 2026-10-25 night, waits for the hub release (v0.171.0): a set-aside-only digest is a warning with „… set aside — nothing failed (<apps>)"; a run with a real failure stays red (`hub/internal/notify/r1287_setaside_test.go`, red-proved) | — | two-repo: the hub words a `unit-set-aside` leg on its own („… a backup of an earlier install was set aside — nothing failed", warning, not error) with a test on the rendered subject; or a new event type through the allow-list (`hub-event-allowlist-gotcha`); one hub release | CC |
|
||||
| **R-1291** | Monitoring & notifications | P3 | **A held app gives the operator no lasting signal: one event at the stop, then nothing — demo-hp's Kimai sat held 11 days unseen.** Found 2026-10-25 night (R-1286). No held-app state reaches the hub (the report carries none; the hub shows none); the crash-loop stop sends `app_stopped_unhealthy` once. On a household box an app held after a crash loop, a failed update or a failed restore stays stopped until someone opens its page. | OPEN | — | a two-repo design: the controller reports its holds (stack, reason, since), the hub shows them on the host and mails a reminder after N days (the D-level of the reason); one row's worth of design first | CC |
|
||||
|
||||
|
||||
## Hub & operator — 13 rows (P3 5, P4 8)
|
||||
|
||||
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-10-26 (morning)
|
||||
|
||||
**Updated 2026-10-25 (evening, attended release).** Every box runs the new controller: a restore that does not start says so and keeps both copies (D49), a reinstall never touches the old backup, a restore downloads first. The hub stores the recovery package in one step; the new agent is vouched; the catalog has three fixes.
|
||||
|
||||
## Moved from STATUS on 2026-10-25 (attended release run)
|
||||
|
||||
**Updated 2026-10-25 (Sunday).** A reinstall keeps old backups whole; a restore downloads first and tells the truth.
|
||||
|
||||
Reference in new issue
Block a user