folder `20261024-220001`, no alarm mail since 2026-10-23 22:20Z). R-1183 — partial today (one hub snapshot file per
slot, `07:00Z` and `13:00Z`; the hub restarted at 15:50Z for 0.170.0, so the morning log is gone); the full read stays
due 2026-10-26.
## What ran, in order
- **Part 1, fix A (R-1270 follow-up):** one helper, „started" = running on every sample of a 45 s window, no
`RestartCount` growth, every healthcheck `healthy`; the undo copy released after 3 more minutes (10 s samples, 30 min
watch; only a restore begun after the watch ends it). Found live and fixed on the way: docker's `-f` template failed on
every app without a healthcheck (now the JSON is read). Live on 9202 rc.6: an image that exits after 20 s → „még nem
indult el", copy kept, one operator line; the original unit back → released 3 min later.
- **Part 2, fix B (R-1279):** the D48 helper before step 1 of the import. Live: a bundle copy with a missing tag →
refused with the D48 sentence, the app running, its volumes untouched.
- **Part 3, hub 0.170.0 (R-1254):** asked → yes → manifest bump, hard refresh, sync of `Deployment/hub` alone,
Synced/Healthy, digest `sha256:6478498d…`. Tester 1 ran the escrow ceremony through the page's endpoints (the new code
stored by the operator as a CC secret); the hub logged „the new package is on the hub". No alarm mail after the
deploy (the mailbox as the second channel; a bind mail to Tester4 arrived, the control).
- **Part 4, agent 0.165.0 (R-1257, R-1250):** `release-agent.sh` from a clean clone; signed jobs to the three boxes
(76/76, the appliance drop-in in force by `sshd -T`). **No BYO host existed, so one was built** on demo-hp: VM 9371, a
plain Proxmox VE 9.2 from an answer-file ISO, Felhom installed with `--mode byo` as Tester4 (the operator deleted the
old host and stored the retrieval passphrase as a CC secret). Our 0.164.0 drop-in planted first; after 0.165.0 still
there; a second config bundle removed it, wrote nothing, `sshd -T` read the owner's own setting. Vouched by the
operator.
- **Part 5, controller 0.343.0:** rc.6/rc.7 on 9202; the set-aside alarm added and red-proved (Part 5.2); `0.343.0`
pushed after CI 2330; per-box floors → the three boxes at 17:17Z; box checks: 0 ERROR lines, all containers up,
https-only green; the D47 count above; on Tester 1 a unit restore and a set-aside restore — the item back, the copy
released after 3m0s each. Asked → yes → global floor 17:53:42Z (MinAgent 0.158.1, read back in the hub log; the BYO
host moved 18 s later); per-box floors cleared; a 30-minute watch: the hub sent 0 alarms (`global-floor-watch.txt`) and no alarm mail arrived (the mailbox as the second channel).
- **Part 6, catalog:** Homepage's family gate tried on the drill on 9202 (stranger 302/401, household 200); asked →
yes → `ebbf1f4` pushed; demo-hp read it 8 minutes later.
- **Part 7, D44/D45 = all picks:** confirm lines tagged, text-follows-code changes made (old words struck through), the
four code picks became R-1280–R-1283; R-1265 and R-1266 closed.
- **R-1250:** released, but its live proof cannot run before the agent's next due window (Tuesday 10:00–15:59 local; all
boxes measured on 2026-10-24) — a dated check on 2026-10-27 with the operator's yes for DooPlex's qBittorrent as peer.
`8dea1f34d77ba40aa44c5461c3728c2ca94bab46854a6bd8b86191f520dfe85a`; agent CI by commit: `c8d3c17` 2306 ✓, `4a1a849` 2325 ✓, `4697f8a` 2326 ✓, `6b6ba0c` 2334 ✗ (R-1290 — the registry, not the code).
- **Controller 0.343.0** `sha256:bb3da72a054c45751e3baf6e4bad43d1561a952717c5f426d65a5aeed0413a3b` from `e2a9761`.
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-10-25 (Sunday).** A reinstall keeps old backups whole; a restore downloads first and tells the truth.
**Updated 2026-10-25 (evening, attended release).** Every box runs the new controller: a restore that does not start says so and keeps both copies (D49), a reinstall never touches the old backup, a restore downloads first. The hub stores the recovery package in one step; the new agent is vouched; the catalog has three fixes.
## What works
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
- A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4).
- 65 catalog apps install, update one tested step at a time, and a failed update undoes itself (map B1, B3, B4, B6).
- Backups: database dumps, a second-drive copy, an off-site copy and whole-box copies; restore tests run by themselves (map C1–C4, A17, A18).
- A restore brings an app back with its data and its database password; a failed or cut restore puts the app back as it was — off-site live, local on the test image (map C5, D6, D16).
- A restore brings an app back with its data and its database password; a failed or cut restore puts the app back as it was; a restore that does not start says so and keeps both copies (map C5, D6, D7, D16).
- After a power cut the box starts by itself, cleans up a half-written backup, and Immich repairs broken photos (map C10, F12).
- A second drive is offered as the backup target; media apps see every drive; a one-disk box gets its own data area (map D1, D13, D15).
- The three test boxes answer only over https; the hub's key opens nothing on a box; password guesses are counted (map E12, E13, E17).
@@ -18,13 +18,12 @@
- The hub mails alarms and recoveries and keeps a weekly whole-box copy on ep0; CC uses its own limited token (map F1, F11, G15).
## What is broken
- **P1 (fixed on main, NOT released):** a power cut while an app on a drive is restored can later pour old data over new data (R-1251); one wrong line in an app's backup settings deletes its second-drive copy (R-1252, P2); three restore holes (R-1269–R-1271).
- **P1:** any hacked app on DooPlex can read the hub's nightly backups, and maybe the signing keys (R-1241, homelab session); about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). The scratch box cannot use agent features (R-1083).
- DooPlex: something sets the shared checkout to „bare" (R-935).
## What we work on
- Dated checks, run by CC. 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes).
- Dated checks, run by CC. 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the qBittorrent line test on a seeding box, with DooPlex's qBittorrent as the peer (R-1250, your yes); the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes).
## Waiting on you — decide
@@ -32,8 +31,6 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|---|---|---|---|---|---|---|
| D49 | R-1270 | A restore's data is back but the app does not start in 90 s. Keep the restored data and wait, or pour the old data back by itself? | **Keep and wait** (a slow first start is common; the old copy is kept, a late start turns it into a success; you get one alarm) | pour back by itself: a slow but correct restore is thrown away | the data stays restored, the old copy stays for support | — |
| D46 | R-1272 | Put Homepage behind the family gate, like Glance? | **Yes** (a stranger now sees the box's CPU, memory and disk, and later the family's links; one line in the template, tried on the drill first) | leave it open: anyone who finds the address sees those | it stays open to strangers | — |
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
@@ -52,10 +49,11 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
**Day, 2026-10-23 — your do list.**
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
2. On the hub's Hosts page: Discard appliance 49 (host Tester4-aa9466 is gone). If not: a dead entry alarms.
2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-f967d4 (the BYO test host, removed 2026-10-25). If not: dead entries alarm.
3. Give the homelab session R-1241, R-1242, the lists at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` and `documentation/audits/dooplex-alarms-2026-10-24/README.md`, and an upload alarm on DooPlex's `eno1` (~100 Mbit/s line; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups.
**The rest**
- System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy.
- Tell CC what removed four old agent packages from Gitea on 2026-10-25 (a clean-up rule?): the agent's CI is red until the gate and the rule agree (R-1290).
- Publish installer 1.36.1 (tag + webpage sync): a wrong passphrase then reads „passphrase REJECTED", not a doubled code (R-1284).
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
- Two DooPlex leftovers: `leftovers/2026-10-14/first-post/` (keep or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). Since 2026-10-14.
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 93fb251 + d582feb (D20, D22)**2026-10-25:** changed again by controller 0.343.0 (D48 pull-first, R-1270 fix A: the verdict needs a stable start, the undo copy goes after 3 more minutes); on 0.343.0 Tester 1's local unit restore and set-aside restore passed and released the copy (`audits/release-2026-10-26/t1-restores-0.343.0/`); this path itself not re-walked. | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
| <a id="d5"></a>[D5](capability-map-detail/D.md#d5) | Network storage (NAS) is browse + bulk-media only — it may NOT host an app's data namespace | controller v0.187.0 | **PROVEN-LIVE** | — | `audits/R108-network-app-namespace-2026-07-30.md` (2026-07-30) |
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb (D22, R-1178) | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
| <a id="d7"></a>[D7](capability-map-detail/D.md#d7) | A restore SAYS what it returned, and refuses what it cannot do — the four restore-surface truth defects from the 2026-08-21 drill | controller v0.226.0 (R-353, R-357, R-358, R-360, R-396) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: three of four live; R-357 implemented only | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (2026-08-30) |
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb (D22, R-1178)**2026-10-25:** changed again by controller 0.343.0 (D48 pull-first, R-1270 fix A: the verdict needs a stable start, the undo copy goes after 3 more minutes); on 0.343.0 Tester 1's local unit restore and set-aside restore passed and released the copy (`audits/release-2026-10-26/t1-restores-0.343.0/`); this path itself not re-walked. | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
| <a id="d7"></a>[D7](capability-map-detail/D.md#d7) | A restore SAYS what it returned, and refuses what it cannot do — the four restore-surface truth defects from the 2026-08-21 drill | controller v0.226.0 (R-353, R-357, R-358, R-360, R-396) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: three of four live; R-357 implemented only**2026-10-25:** controller 0.343.0 adds two truths here — a restore whose app does not start says so and keeps its copy (R-1270, D49 = A; live on 9202 rc.6), and a `.fab` import that cannot download its version refuses before it touches the app (R-1279, live). | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (2026-08-30) |
| <a id="d8"></a>[D8](capability-map-detail/D.md#d8) | The box PROVES its own off-site copy still HOLDS something — a backup that is intact and EMPTY is caught without a person | controller v0.231.0 (R-87) + hub v0.110.0 | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r87-offsite-proof-2026-08-31/` (2026-08-31) |
| <a id="d9"></a>[D9](capability-map-detail/D.md#d9) | The off-site store is VERIFIED on a cadence — something checks that the customer's backups are still readable | controller v0.228.0 (R-359, R-397, R-399) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r359-integrity-2026-08-30/` (2026-08-30) |
| <a id="d13"></a>[D13](capability-map-detail/D.md#d13) | Media apps see EVERY drive; a drive added, pulled, back or safely removed is handled per app (notice, real restart, wiring, torrents paused/resumed) | controller v0.313.0, catalog `0188430`, hub v0.151.0 | **PROVEN-LIVE** | Tier-0 boxes (Tester 1, demo-hp 9201); the fleet with golden 0.313.0 + the floor | `audits/media-build-2026-10-15/G/` (2026-10-15); `audits/media-drive-events-2026-10-14/` (2026-10-14) |
| <a id="d14"></a>[D14](capability-map-detail/D.md#d14) | A second local copy COUNTS only on a proven different physical disk: the agent knows each place's disks; a VM box makes the copy but says it cannot check; a same-disk copy is not made; a NAS never hides the one-copy notice; a box with no drive refuses a data-folder app in words | agent v0.157.0, controller v0.317.0, hub v0.154.0 | **PARTIAL** | live: VM `unknown` (Tester 1) and two real-disk boxes `proven` (demo-hp, N100); unit only: the same-disk skip, the NAS notice, the no-drive sentence; `usb_bridge` never seen on a device | `audits/storage-identity-buildA-2026-10-15/README.md` (2026-10-15); `architecture/16-storage-physical-identity.md` §13 |
| <a id="d15"></a>[D15](capability-map-detail/D.md#d15) | A one-disk box gets its own data area („Belső tárhely") outside every whole-box copy, registered as the default drive with no press, so media and data-folder apps install with no extra step; a free disk's partitions can be formatted or used one by one | installer 1.33.0, agent v0.158.1, controller v0.318.0, golden 0.318.0 | **PROVEN-LIVE** | fresh 512 GB one-disk VM on demo-hp; partitions on a scratch disk; the old-box signed job and the D4 sentence unit-proven only; an empty part asks for confirmation (R-1093) | `audits/storage-identity-buildB-2026-10-15/README.md` (2026-10-16) |
| <a id="d16"></a>[D16](capability-map-detail/D.md#d16) | A failed or cut restore puts the household's data back EXACTLY as it was and starts the app — local unit, second drive, kept load and off-site, and after a power cut or a killed controller (D22) | controller `main` (0.338.0-rc.1), unreleased | **IMPLEMENTED** | proven live on 9202 (rc.1): a damaged unit and a controller killed between the volume step and the replay; the second-drive pre-check (R-1178), the mirror skip (R-1180) and the off-site power cut by test only; a database outside the replaced volumes is not covered | `audits/night-2026-10-21/P3-restore/02-run.log` (2026-10-21) |
| <a id="d16"></a>[D16](capability-map-detail/D.md#d16) | A failed or cut restore puts the household's data back EXACTLY as it was and starts the app — local unit, second drive, kept load and off-site, and after a power cut or a killed controller (D22) | controller 0.338.0 (D22) → **0.343.0** (released 2026-10-25: R-1251 a copy on an unbound drive waits; fix A) | **IMPLEMENTED** | proven live on 9202 (rc.1): a damaged unit and a controller killed between the volume step and the replay; the second-drive pre-check (R-1178), the mirror skip (R-1180) and the off-site power cut by test only; a database outside the replaced volumes is not covered**2026-10-25 (re-check):** released in 0.343.0; the release run proved the success path's copy release on Tester 1, not a failure put-back on the release build — the status stays. | `audits/night-2026-10-21/P3-restore/02-run.log` (2026-10-21) |
@@ -750,7 +750,11 @@ the snapshot added is removed, the live definition is written back and the app i
restore failed and nothing changed (with placed files: its database and volumes are back, the copied files stay); the
operator gets `backup_run_failures` leg `restore-put-back`. The hold (`restore_mixed`) remains only for a put-back that
fails — then the copy is kept for support — and the double failure (replay AND rollback) holds as before. On success the
copy is removed. **`[FACT]` 2026-10-20 (9202, controller 0.336.0-rc.1, `audits/night2-2026-10-20/P2/live/`):** the same
copy is removed — **since controller 0.343.0 (fix A, 2026-10-25) only after a STABLE start, and the verdict and the release
are two different tests (§8.2e):** the verdict „started" needs the stack running on every sample of a 45 s window, no
container restart in it and every healthcheck `healthy`; the copy is released only after a further 3 minutes of the same
(10 s samples, watched up to 30 minutes; a new restore of the app ends the watch and leaves the copy to it). A restore that
passes the verdict and then flaps keeps its copy. **`[FACT]` 2026-10-20 (9202, controller 0.336.0-rc.1, `audits/night2-2026-10-20/P2/live/`):** the same
forced failure as on 2026-10-19 → 36 tables, the backup-only table absent, schema/data/row-count md5 equal to before,
the app running, no hold, the copy (69 MB, 0.9 s) gone; a normal restore of the same snapshot then worked. The R-893
known limit above is closed for named volumes; placed FILES are still not taken back (the plain-files mix, as ruled).
@@ -1726,7 +1730,7 @@ FOLDER (`${HDD_PATH}` binds under `appdata/`); a volume app keeps nothing there
volumes and keeps only its backups — so the install starts empty, and the kept backup is now the set-aside unit on the
restore page. (The missing offer at install is its own row, R-1278.)
### 8.2e A restore whose app does not start is never „success" [DESIGN, R-1270, 2026-10-25 — controller `b680db8`, unreleased, rides v0.343.0]
### 8.2e A restore whose app does not start is never „success" [DESIGN, R-1270, 2026-10-25 — controller `b680db8` + fix A; D49 = A (operator 2026-10-25); released in controller 0.343.0]
**[FACT] Measured before:** the health check after a restore was a WARN on every path, so a Tandoor restore whose app
stayed `starting` was recorded `ok: true`, „…visszaállítva — az alkalmazás újraindult." (catalog sweep 2026-10-24; 9202
@@ -1742,6 +1746,16 @@ put-back leaves (§8 D20/D22), so no new retention rule. A watcher looks 30 more
turns to success („…elindult: a visszaállítás kész.") and the copy is removed, as after any successful restore; if it
never runs, the copy stays for support.
**D49 = A (operator ruling 2026-10-25, attended release run; `09` §3 decision 258):** the safest reading above is the rule —
the box keeps the restored data, keeps the old copy, tells the operator, and never puts the old data back by itself.
**Fix A (controller 0.343.0, R-1270 follow-up):** „started" is one helper on every path — the stack running on every
sample of a 45 s window, no container's `RestartCount` grown in it, every container with a healthcheck `healthy` (a
container without one is judged by running alone). An app that starts and then crash-loops is „not started". The undo
copy's RELEASE is a separate, longer test: 3 minutes of the same after the verdict (10 s samples, watched up to 30
minutes). **[FACT]** 9202 rc.6: an image that exits after 20 s → not started, copy kept, one operator line; the original
unit back → success, copy released 3 minutes later. Tester 1 on 0.343.0: a unit restore and a set-aside restore, each
„ran stably for 3m0s … the restore copy is released" (`audits/release-2026-10-26/`).
### 8.2c Decisions in code that no document recorded [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
Read at `file:line` on `main` (controller `321df1f`, hub `44d19042`) by the main session:
backup-target-apply, os-apply and priv-apply wrappers; the crash guard; the config-bundle record
- pveum: the Felhom roles/user/token/scoped ACL; the emptied felhom pool
- the install state file
KEPT (lives on deliberately — remove/rotate these out-of-band if the customer is leaving):
- the enrolled drives + ALL data under /mnt/felhom-drives — unmounted only, NEVER wiped;
the drives are physically removable now.
- the PBS backups + this customer's namespace on the PBS side — delete there if wanted
- the hub host/customer record + report history (operator UI / DB)
- this host's WireGuard PEER on the hub and the off-site endpoint (its /32 and public key) — the
tunnel is down from this side; delete the peer in the hub (operator) if the customer is leaving
- the escrow blob in the hub, if one exists (operator UI)
[0;32m[OK][0m UNINSTALL complete — removed: guest 9201, the felhom-agent (unit/sudoers/binary/state/config+baks/user + selfupdate-artifacts/shared-parent/mkfs-wrapper/pbs-apply-wrapper/hook-snippet/dnsmasq-snippets), the pveum role/user/token/ACL, the felhom pool, and /var/lib/felhom-install/state.json.
[0;32m[INFO][0m golden vzdump: left in place (--remove-golden to remove).
[0;32m[INFO][0m NOTE: the 'sudo' package was left INSTALLED (a system package, never ours to remove). dnsmasq is removed only when the install-time record says Felhom installed it; a resolver that pre-dated Felhom is never touched. The host record still exists in the hub — remove it there if desired.
"line":"2026/10/25 17:29:15 [INFO] [backup] tandoor: ran stably for 3m0s after its unit restore — the restore copy is released (R-1270)",
"undo_left":""
},
"reinstall":true,
"set_aside_line":"2026/10/25 17:31:55 [INFO] [backup] tandoor: the recovery unit of an earlier install (795457b5d5a73a21; the install ids differ) is SET ASIDE whole → /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T173155Z; this install (c150b1bb5020980f) gets its own unit from its first backup (D47)",
"line":"2026/10/25 17:37:20 [INFO] [backup] tandoor: ran stably for 3m0s after its unit restore — the restore copy is released (R-1270)",
"undo_left":""
},
"controller_lines":"2026/10/25 17:24:33 [WARN] [web] Restore requested (async): stack=tandoor, snapshot=helyi from 172.18.0.3:46404\n2026/10/25 17:24:33 [INFO] [backup] Restore tandoor: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)\n2026/10/25 17:24:33 [INFO] [backup] Restoring tandoor from recovery unit /mnt/sys_drive/felhom-data/backups/primary/tandoor: images=2, secrets recovered=2/2, data_keys=0\n2026/10/25 17:24:38 [INFO] [offbox] tandoor: the live volumes are copied before the restore (3 copied, 0 new from the snapshot) in 2.712s — /mnt/sys_drive/felhom-data/backups/restore-undo/tandoor/20261025T172435Z (R-893)\n2026/10/25 17:24:38 [INFO] [backup] tandoor: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195)\n2026/10/25 17:24:38 [INFO] [backup] Restoring Docker volume tandoor_tandoor_media for tandoor\n2026/10/25 17:24:38 [INFO] [backup] Restoring Docker volume tandoor_tandoor_postgres_data for tandoor\n2026/10/25 17:24:40 [INFO] [backup] Restoring Docker volume tandoor_tandoor_static for tandoor\n2026/10/25 17:24:42 [INFO] [backup] Restored 3 Docker volume(s) for tandoor\n2026/10/25 17:24:43 [INFO] [backup] Restore tandoor: replaying DB dump into tandoor-postgres (postgres)\n2026/10/25 17:24:46 [INFO] [backup] Restore tandoor: replayed 1 DB dump(s)\n2026/10/25 17:26:13 [INFO] [backup] Restore-from-unit completed: tandoor — 3 volume(s) of 3 listed, 1 database(s) of 1 listed\n2026/10/25 17:26:13 [INFO] [web] Restore completed (async): stack=tandoor in 1m40.039853575s (volumes 3/3, dbs 1/1)\n2026/10/25 17:29:15 [INFO] [backup] tandoor: ran stably for 3m0s after its unit restore — the restore copy is released (R-1270)\n2026/10/25 17:29:56 [INFO] [stacks] image retention (remove of tandoor): pass over 24 image(s) of [ghcr.io/tandoorrecipes/recipes postgres] — 2 candidate(s), 2 deleted, the rest kept\n2026/10/25 17:31:55 [INFO] [settings] kept copy recorded: kind=unit source=\"tandoor\" reason=earlier_install path=/mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T173155Z\n2026/10/25 17:31:55 [INFO] [backup] copy KEPT (earlier_install): unit \"tandoor\" at /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T173155Z\n2026/10/25 17:31:55 [INFO] [backup] tandoor: the recovery unit of an earlier install (795457b5d5a73a21; the install ids differ) is SET ASIDE whole → /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T173155Z; this install (c150b1bb5020980f) gets its own unit from its first backup (D47)\n2026/10/25 17:31:55 [INFO] Event pushed: backup_run_failures (error) [hu-only] — App \"tandoor\": its backup belonged to an earlier install (795457b5d5a73a21; this install c150b1bb5020980f), so it was SET ASIDE whole at /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T173155Z (D47) and \n2026/10/25 17:32:39 [WARN] [web] Restore of tandoor from a set-aside unit requested (async): /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T173155Z from 172.18.0.3:49058\n2026/10/25 17:32:39 [INFO] [backup] restore tandoor from the set-aside unit /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T173155Z (an earlier install, D47)\n2026/10/25 17:32:39 [INFO] [backup] Restore tandoor: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)\n2026/10/25 17:32:39 [INFO] [backup] Restoring tandoor from recovery unit /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T173155Z: images=2, secrets recovered=2/2, data_keys=0\n2026/10/25 17:32:48 [INFO] [offbox] tandoor: the live volumes are copied before the restore (3 copied, 0 new from the snapshot) in 4.781s — /mnt/sys_drive/felhom-data/backups/restore-undo/tandoor/20261025T173243Z (R-893)\n2026/10/25 17:32:48 [INFO] [backup] tandoor: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195)\n2026/10/25 17:32:48 [INFO] [backup] Restoring Docker volume tandoor_tandoor_media for tandoor\n2026/10/25 17:32:48 [INFO] [backup] Restoring Docker volume tandoor_tandoor_postgres_data for tandoor\n2026/10/25 17:32:50 [INFO] [backup] Restoring Docker volume tandoor_tandoor_static for tandoor\n2026/10/25 17:32:53 [INFO] [backup] Restored 3 Docker volume(s) for tandoor\n2026/10/25 17:32:53 [INFO] [stacks] tandoor: the restored data belongs to install 795457b5d5a73a21 (was c150b1bb5020980f) — its backups follow it (D47)\n2026/10/25 17:32:53 [INFO] [backup] Restore tandoor: replaying DB dump into tandoor-postgres (postgres)\n2026/10/25 17:32:57 [INFO] [backup] Restore tandoor: replayed 1 DB dump(s)\n2026/10/25 17:34:18 [INFO] [backup] Restore-from-unit completed: tandoor — 3 volume(s) of 3 listed, 1 database(s) of 1 listed\n2026/10/25 17:36:54 [INFO] [settings] kept copy recorded: kind=unit source=\"tandoor\"reason=earlier_installpath=/mnt/sys_drive/felhom-data/backupLine truncated
"after_remove":{
"containers":"",
"volumes":"",
"appyaml":"",
"appdata":"",
"kept":"",
"subdomain":"404"
},
"set_aside_after_remove":"ls: cannot access '/mnt/*/felhom-data/backups/set-aside/tandoor': No such file or directory"
17:24:33Z [6] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
17:26:13Z [1] unit restore: {"running": false, "op": "restore", "stack": "tandoor", "started_at": "2026-10-25T17:24:33.058022807Z", "last": {"op": "restore", "stack": "tandoor", "ok": true, "message": "A(z) tandoor: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-25T17:26:13.1001
17:26:43Z tandoor: readback of the seeded account found=True
17:26:43Z [1] item after the unit restore: True
17:29:19Z [1] released: 2026/10/25 17:29:15 [INFO] [backup] tandoor: ran stably for 3m0s after its unit restore — the restore copy is released (R-1270) | undo folder now: gone
17:32:30Z [2] set aside: 2026/10/25 17:31:55 [INFO] [backup] tandoor: the recovery unit of an earlier install (795457b5d5a73a21; the install ids differ) is SET ASIDE whole → /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T173155Z; this install (c150b1bb5020980f) gets its
17:32:39Z [2] POST /backup/kept/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
17:34:49Z tandoor: readback of the seeded account found=True
17:34:49Z [2] item after the kept-data restore: True
17:37:25Z [2] released: 2026/10/25 17:37:20 [INFO] [backup] tandoor: ran stably for 3m0s after its unit restore — the restore copy is released (R-1270) | undo folder now: gone
17:24:33Z [6] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
17:26:13Z [1] unit restore: {"running": false, "op": "restore", "stack": "tandoor", "started_at": "2026-10-25T17:24:33.058022807Z", "last": {"op": "restore", "stack": "tandoor", "ok": true, "message": "A(z) tandoor: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-25T17:26:13.1001
17:26:43Z tandoor: readback of the seeded account found=True
17:26:43Z [1] item after the unit restore: True
17:29:19Z [1] released: 2026/10/25 17:29:15 [INFO] [backup] tandoor: ran stably for 3m0s after its unit restore — the restore copy is released (R-1270) | undo folder now: gone
17:32:30Z [2] set aside: 2026/10/25 17:31:55 [INFO] [backup] tandoor: the recovery unit of an earlier install (795457b5d5a73a21; the install ids differ) is SET ASIDE whole → /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261025T173155Z; this install (c150b1bb5020980f) gets its
17:32:39Z [2] POST /backup/kept/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
17:34:49Z tandoor: readback of the seeded account found=True
17:34:49Z [2] item after the kept-data restore: True
17:37:25Z [2] released: 2026/10/25 17:37:20 [INFO] [backup] tandoor: ran stably for 3m0s after its unit restore — the restore copy is released (R-1270) | undo folder now: gone
## 2026-10-25 — the attended release run (full text of the rows: `git show 9647f1d1:documentation/backlog/OPEN-ITEMS.md`)
## 2026-10-25 — the attended release run (full text of the rows: R-1265, R-1266`git show 9647f1d1:documentation/backlog/OPEN-ITEMS.md`; the release closures `git show 36ef8dcd:documentation/backlog/OPEN-ITEMS.md`)
| Row | What | Closed | Evidence |
|---|---|---|---|
| **R-1251** | **A power cut during a restore of a DRIVE-backed app: the app then runs on the cut restore's data, and a later controller restart pours the pre-restore copy back over everything written since — the D22 promise is false for every drive-backed app.** (P1) | CLOSED 2026-10-25 — released in controller **0.343.0** (image `sha256:bb3da72a…`, from `e2a9761`, CI 2330 ✓; rc.6/rc.7 on 9202; per-box floors on Tester 1, demo-hp, demo-felhom 17:16Z, box checks + https-only green; global floor 17:53:42Z, MinAgent 0.158.1, read back on the hub log). Fix on main since `bb2809d` + `062f9b4` (red-proved, restore-holes brief); the put-back waits for an unbound drive. | `audits/release-2026-10-26/` |
| **R-1252** | **One rejected backup block in an app's `.felhom.yml` makes that night's second-drive copy DELETE the app's copies of the household's files.** (P2) | CLOSED 2026-10-25 — released in controller **0.343.0** (image `sha256:bb3da72a…`, from `e2a9761`, CI 2330 ✓; rc.6/rc.7 on 9202; per-box floors on Tester 1, demo-hp, demo-felhom 17:16Z, box checks + https-only green; global floor 17:53:42Z, MinAgent 0.158.1, read back on the hub log). A rejected backup block never deletes second-drive copies (red-proved, restore-holes brief). | `audits/release-2026-10-26/` |
| **R-1253** | **An export with „stop the app first” during another app's restore overwrites the ONE app-stop marker file — the restore's crash record and its Undo are lost; a power cut then leaves the restored app on mixed data with no put-back.** (P2) | CLOSED 2026-10-25 — released in controller **0.343.0** (image `sha256:bb3da72a…`, from `e2a9761`, CI 2330 ✓; rc.6/rc.7 on 9202; per-box floors on Tester 1, demo-hp, demo-felhom 17:16Z, box checks + https-only green; global floor 17:53:42Z, MinAgent 0.158.1, read back on the hub log). An export waits for a running restore (409) (red-proved). | `audits/release-2026-10-26/` |
| **R-1269** | **A reinstall's own capture rewrites the unit that „remove — keep backups” kept, so restoring that unit breaks the app and the household's item is gone.** (P2) | CLOSED 2026-10-25 — released in controller **0.343.0** (image `sha256:bb3da72a…`, from `e2a9761`, CI 2330 ✓; rc.6/rc.7 on 9202; per-box floors on Tester 1, demo-hp, demo-felhom 17:16Z, box checks + https-only green; global floor 17:53:42Z, MinAgent 0.158.1, read back on the hub log). D47: on upgrade every installed app got an install id (Tester 1 3/3, demo-felhom 5/5, demo-hp 11/12 — kimai held since 10-14, R-1286), 0 units set aside; on Tester 1 a reinstall set the earlier unit aside, the operator got the mail (17:31:56Z), the set-aside restore brought the item back. The set-aside now raises an operator event (red-proved, `redproof/d47-setaside-notify.txt`). Follow-ups: R-1287, R-1288. | `audits/release-2026-10-26/` |
| **R-1270** | **A unit restore reports success while the app never came up.** (P2) | CLOSED 2026-10-25 — released in controller **0.343.0** (image `sha256:bb3da72a…`, from `e2a9761`, CI 2330 ✓; rc.6/rc.7 on 9202; per-box floors on Tester 1, demo-hp, demo-felhom 17:16Z, box checks + https-only green; global floor 17:53:42Z, MinAgent 0.158.1, read back on the hub log). Fix A (D49 = A, operator 2026-10-25): the verdict needs a 45 s stable window; the undo copy goes only after 3 min stable. Live on 9202 (rc.6): an app that exits after 20 s → „még nem indult el”, copy kept, one alarm; a normal restore → released after the window. On Tester 1 (0.343.0): a unit restore and a kept-data restore, each released after 3m0s (`t1-restores-0.343.0/`). Red-proofs `redproof/fixA-*.txt`. | `audits/release-2026-10-26/` |
| **R-1271** | **„Use my kept data” loaded ANOTHER box's database — a future-dated off-site copy of the rehearsal box — under today's kept files.** (P2) | CLOSED 2026-10-25 — released in controller **0.343.0** (image `sha256:bb3da72a…`, from `e2a9761`, CI 2330 ✓; rc.6/rc.7 on 9202; per-box floors on Tester 1, demo-hp, demo-felhom 17:16Z, box checks + https-only green; global floor 17:53:42Z, MinAgent 0.158.1, read back on the hub log). A kept copy > 10 min in the future is never the newest (proven on a forged copy, restore-truth run). | `audits/release-2026-10-26/` |
| **R-1279** | **The `.fab` import removes a running app's volumes before its images are pulled — D48 does not cover it.** (P3) | CLOSED 2026-10-25 — released in controller **0.343.0** (image `sha256:bb3da72a…`, from `e2a9761`, CI 2330 ✓; rc.6/rc.7 on 9202; per-box floors on Tester 1, demo-hp, demo-felhom 17:16Z, box checks + https-only green; global floor 17:53:42Z, MinAgent 0.158.1, read back on the hub log). Fix B: the `.fab` import downloads first (the D48 helper before step 1); live on 9202 (rc.3, again rc.6): a bundle with a missing tag refused with the D48 sentence, containers and volumes untouched (`fixprobe-run.txt`, `redproof/fixB-no-prepull.txt`). | `audits/release-2026-10-26/` |
| **R-1257** | **D4's root-key-only sshd drop-in reaches BYO hosts too, while `11` §1 says a BYO host gets guest and Docker updates only — never a host change — and the installer's BYO screen lists what it leaves alone.** (P2) | CLOSED 2026-10-25 — agent **0.165.0** released (`release-agent.sh` from a clean clone; binary `e9d04a47…`, bundle `8dea1f34…`), signed `agent_update` to Tester 1, demo-hp, demo-felhom (self-check 76/76; appliance drop-in in force, `sshd -T`), and to a BYO host built for it (VM 9371 on demo-hp, `Tester4-f967d4`): nothing written, our old drop-in removed — on the SECOND bundle run (R-1285). **Vouched by the operator 2026-10-25 18:38:49 CET** (`Artifact manifest set: agent=0.165.0 … bundle_sha=8dea1f34…`). | `audits/release-2026-10-26/` |
| **R-1272** | **Homepage answers a stranger with no gate at all** (P3) | CLOSED 2026-10-25 — **D46 = yes** (operator); catalog `ebbf1f4` (`family_gate: true`, `min_controller: 0.287.0`), tried on the drill on 9202 first (stranger 302 to `/__family/start` / 401, household 200, `homepage-gate-9202.txt`), pushed on the operator's yes 17:39Z; demo-hp's catalog read it at 17:47:20Z. | `audits/release-2026-10-26/` |
| **R-1265** | **Decisions found in code with no document — the operator confirms them (review round 2).** (P3) | CLOSED 2026-10-25 — **D44 = all picks** (operator, attended release run): the confirm lines carry a normal `[DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]` (`07` §8.2c, `09` §6.4.x, `14`, `15`, `16`); `03` §6 lists the 26 unnamed agent routes; `07` §8 gains row 16 (the registry lost with DooPlex, RTO blank and listed in §8.1); the four code-change picks became rows **R-1280–R-1283** (no code changed) | `audits/review-round-2-2026-10-24/D44-D45-picks.md`; the architecture diffs of this commit |
| **R-1266** | **Rulings and promises in the documents that the code does not do as written.** (P3) | CLOSED 2026-10-25 — **D45 = all picks**: the text follows the code, each change tagged `[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]`, old words struck through, not deleted (`09` decisions 13, 14, 250 item 5; `01` the poke and the operator door, the locked packages with both limits named; `03` §2/§6 break-glass; `06` the two tunnel-side listeners, the withdrawn plan) | the architecture diffs of this commit |
| **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)****Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator |
| **R-1284** | Install & onboarding | P3 | **The installer's preflight never words a refusal: `http_code` printed the status twice („401401"), so the 401 („passphrase REJECTED"), 404 („customer not found") and 409 (R-1167's live-box sentence) branches never matched — every refusal read „unexpected hub status 401401".** Seen live 2026-10-25 on the BYO test VM (attended release run) with installer 1.36.0: a wrong id read `404404`, a wrong passphrase `401401`. R-1167 was CLOSED 2026-10-20 on a publish whose 409 sentence could not show. With `-f`, curl prints `-w` and exits 22 on a 4xx, and the fallback curl printed the code again (line 1833 already worked around it by keeping the last 3 characters). | OPEN — **fixed on main, NOT published:** installer 1.36.1 (`http_code`: one curl without `-f`, never failing); `test_http_code_prints_one_status_on_a_refusal`, seen failing on the old helper (`audits/release-2026-10-26/redproof/installer-http-code-doubled.txt`). | the publish is a DooPlex change (tag + webpage git-sync pins + the webpage sync) — the operator's | publish installer 1.36.1 (cut `installer-v1.36.1`, bump both `--ref` in `manifests/webpage.yaml`, sync the webpage alone); then a preflight with a wrong passphrase reads „passphrase REJECTED (401)"; then close | operator |
## Apps & catalog — 17 rows (P3 7, P4 10)
## Apps & catalog — 16 rows (P3 6, P4 10)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1091** | Apps & catalog | P3 | **Catalog copy that is wrong or stale at the pinned tag, found writing the guides — each needs an operator word or a live check, so none was changed tonight.** From source at each app's tag (evidence per app in `audits/app-guides-2026-10-15/B/`): (1) and (2) **DONE 2026-10-16** (operator decision D12: reword): calcom's use case now names Apple Calendar or a CalDAV calendar instead of Google/Outlook, and bentopdf's tagline and use case say the browser does the work and the file stays on the device (catalog CHANGELOG 2026-10-16); (3) **sabnzbd**, **nzbget** use_cases say Radarr/Sonarr/Jellyseerr hand them jobs — the box wires only qBittorrent and Transmission (`mediawire.go`), and a hand link needs a path mapping; (4) **onlyoffice** first_steps 3–4 tell the household to wire Nextcloud by hand — the integration toggle does it (`internal/integrations/onlyoffice_nextcloud.go`); (5) **glance**, **homepage** first_steps send the household into config files in a Docker volume no household tool reaches; (6) **lubelogger** starts with only the „Dashboard" tab and in miles (shipped `appsettings.json`), so first step 3 cannot be done as written; (7) **home-assistant** first_steps promise auto-discovered devices — the app runs on a Docker bridge, where mDNS/SSDP were measured deaf (spike R-6); (8) **homebox** and **gitea**`add_people` name menus that moved or read differently at the tag; **bookstack**`default_creds` still says `admin@admin.com / password`, which after_install replaces; **dawarich** names the API key's page „Settings" (source suggests the account page); **code-server** says „install your tools" with no sudo; **komga** names the discontinued Tachiyomi; **seerr**`docs_url` points at Overseerr; **wanderer**'s comment still says it cannot close sign-up (R-714 is fixed in the same file); (9) **nzbget** and **grafana** deploy-field descriptions say the password can be changed later — nzbget re-applies its env password at every start and grafana ignores the field after first start (R-1086 is the qBittorrent/Transmission side). | OPEN — filed 2026-10-15 | — | Live-check (5)–(7) and (9) on 9202; fix (3), (4), (8) as copy with the freeze re-admitted; re-run the guide label check for any app whose copy moves | CC |
| **R-1272** | Apps & catalog | P3 | **Homepage answers a stranger with no gate at all**, while Glance — the same class of dashboard — is behind the family gate (decision 221, S26). A stranger who reaches `homepage.<domain>` sees the box's CPU, memory and disk figures and, once the family edits it, their own links (`audits/catalog-sweep-2026-10-24/homepage/`; the stranger request → 200). | WAITING-ON-OPERATOR | the operator's word | Decide: `family_gate: true` like Glance (CC's pick: one line in the template, proven on the drill first) or leave it open | operator |
| **R-1092** | Apps & catalog | P4 | **Three apps show no guide: plant-it and home-assistant carry one HIDDEN (`checked_version: ""`), emby has none.** plant-it: `lifecycle: abandoned`, and its pinned image does not exist upstream, so there is no screen to check labels against. home-assistant: its frontend follows the browser language and the Hungarian strings are fetched at build time, not kept in git, so the Hungarian labels cannot be checked against source (the draft is written; the shipped frontend in the image could check them). emby: its server's web UI is in no public source at 4.11.0.4. Evidence: `audits/app-guides-2026-10-15/B/`. The other 62 templates show a checked guide (controller ≥ 0.315.0). | OPEN — filed 2026-10-15 | — | home-assistant: check the Hungarian labels against the frontend files in the pinned image (read-only, on a box running it), then set checked_version; emby: label-check against the image's own web files the same way; plant-it: nothing while abandoned | CC |
| **R-676** | Apps & catalog | P3 | **[P3-LOW] Watch: immich's first start restarted 12 times — decision 28's crash-loop stop (6 in 10 min) would stop it.** From the 2026-09-17 chaos night (DB connection dropped during the first-start geocoding import on a 6 GB guest; it did not recover that night). No healthy app in any drill evidence restarts on a first start (1831 samples, 40 live containers), so the threshold stands; this row exists so the first immich install under v0.269.x is watched. `audits/night-2026-09-24/A3/40-first-start-restarts.txt`**2026-09-25 night (read from source, v0.271.0): a DEPLOY's first start is NOT covered by decision 28's suppression** — `Deploying` clears when `compose up -d` returns (`deploy.go` "Clear deploying flag"), and `ObserveUnhealthy` then samples the app; an automatic update's step, verify and undo ARE covered (`Updating`, pinned by `TestD28_NoCrashLoopStopDuringAnAutomaticStep`). So a first start that restarts ≥ 6 times in 10 min is stopped — which R-676 already accepts for a broken first start; a healthy slow first start would be stopped too. **-- 2026-09-30: the first-start restarts are explained.** immich's first-start geodata import OOM-kills its database at 512M on a guest with no swap (R-732, measured: 61–104 kills); the 2026-09-17 chaos-night case (DB connection dropped during the import on a 6 GB guest) fits it. Fixed in the catalog (`56c4888`, 768M). The watch itself (decision 28 on a DEPLOY's first start) is unchanged. | **OPEN — P3; owner: CC (watch)** | — | — | CC |
| **R-1028** | Apps & catalog | P3 | **Plex's claim code can run out during the install, and the box cannot recover it.** plex.tv's code lives 4 minutes from the moment the household copies it; the clock covers the wizard, the image pull (`plexinc/pms-docker` is 351 MB, measured) and the first start. Plex reads `PLEX_CLAIM` only on its first run, and the field is `locked_after_deploy` — so a late code leaves an unclaimed server and the only way back is remove + install again. Reported by the operator during the 2026-10-14 test; on Tester 1's line the pull+start took 8.5 s and the claim worked (`audits/media-all-drives-2026-10-14/``G-auth/G3-plex-claim.txt`), so it bites on a slow line or a slow hand. **K6, 2026-10-14 night (`audits/media-drive-events-2026-10-14/``K6-1-unclaimed-plex.txt`):** an unclaimed Plex starts with no code at all, and `POST /myplex/claim?token=` on the RUNNING server reaches plex.tv (a wrong code → 500 after plex.tv's 401). So the box can pull + start first and ask for the code after: no race. A real claim needs the household's code — the morning step on the throwaway `plex-k6` on Tester 1. Narrowed 2026-10-14 night. **PROVEN 2026-10-15 morning (`K6-2-claim-after-start.txt`):** the operator's real code, given to the RUNNING unclaimed `plex-k6` → 200, `claimed` 0→1, `PlexOnlineToken` written, and the open door closed (no token → 401). `ADVERTISE_IP` was unset and the claim did not need it; whether clients find the server without it was not measurable (the throwaway had no published port). **BUILT 2026-10-15 (v0.313.0):**`after_start` field — not asked at install; the Plex page asks for the code once Plex runs and claims the running server; the gate opens only when Plex says claimed. Proven up to the code on 9201 (G5). | **NARROWED** — the dashboard claim with a real code is the last step | a household Plex code | Claim through the Plex page with a real code (G5c); then close | CC |
@@ -159,20 +158,14 @@ stopping line that lies.
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1251** | Backup & restore | P1 | **A power cut during a restore of a DRIVE-backed app: the app then runs on the cut restore's data, and a later controller restart pours the pre-restore copy back over everything written since — the D22 promise is false for every drive-backed app.** Found 2026-10-24 (review round 2, HUNT-1 H1). Read at `file:line` (controller `321df1f`): the undo copy lives on the app's drive (`internal/backup/offbox_restore_undo.go:47-48`); `AppStopGuard.Recover` runs at start BEFORE the drive is bound, finds the copy „gone” and restarts „as before” (`appstop_marker.go:362-367`); the gated start refuses (drive absent) → marker and Undo KEPT (`:383-387,408-411`); the drive binds and the boot recreate starts the app checking only „held” (`internal/web/intermediary.go:470-476`); the household writes; the next controller start (e.g. the nightly self-update) finds the copy present and pours it back. **Proven with the real `Recover` code** in a lean clone (`audits/review-round-2-2026-10-24/``proofs/H1-drive-cut-restore/`); NOT reproduced on 9202 (its drive is mounted at boot and its agent API is off, so the recreate never runs there). Same family: H2 — a FAILED crash put-back leaves the app stopped and unheld, so Start or the recreate runs it and a later restart pours an old copy back (`appstop_marker.go:368-371`; `api/router.go:690-709`); H7 — Stop pressed during a restore skips the put-back after a cut (`appstop_marker.go:356-361`). **Fixed on main 2026-10-24 night** (controller `bb2809d` + `062f9b4`; restore-holes brief Part 1): a copy on an unbound drive WAITS (restore hold `putback_pending`), every start path honours the hold, the put-back runs first when the drive binds; H2 a failed put-back holds (D20); H7 Stop still gets the put-back; the copy's folders are synced before the marker names it (run 1 of the live proof found `undo.json.tmp` after the cut). **Proven live on Tester 1** (`audits/restore-holes-2026-10-24/P1-tester1/`). | **VERIFY** — fixed on main, waits for the release | the next controller release | release the controller (Monday list); re-read P1's two observables on the released image (a waiting hold at start, the put-back before the start) | CC |
| **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | NARROWED — **2026-10-23 (operator yes D33, D34):** the hub DB now leaves every 6 hours (hub 0.167.0 + `scripts/hub-db-backup/`, alarm at 13 h; restored back byte-identical) and the released images and agent packages leave nightly (`scripts/release-copy/`, R-1227 closed). (c)–(g) unchanged. **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/`**LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest **2026-10-24 (attended): `DooplexGiteaOffsiteStale` fired from 00:57Z — not this path's fault.** The 00:20 push refused, as designed, a 12 h old Gitea dump: the PostgreSQL dump CronJob had failed every run since 2026-10-23 16:00Z (`28P01`) because the window reboot restarted the database and CNPG put the bootstrap secret's password in force (homelab `531023d`, operator's pick: the CronJob only). Fixed; dump, push and restore test run by hand (26 456 files, 12 repos `git fsck`, `gitea.dump` 1114 entries); cleared 10:51Z (`audits/dooplex-alarms-2026-10-24/`). Backup alarms now arrive as ONE mail every 4 h (homelab `22d7697`). **2026-10-23 night — survey again, read only (`audits/dooplex-survival-2026-10-22/PLAN.md`):** both nightly off-site jobs Line truncated
| **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):**`_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). **FIRST VERDICT READ 2026-10-21 (rehearsal session, read-only):** the 07:10 CEST run judged three tiers against yesterday — demo-felhom 5, demo-hp 14, tester-1 4 new snapshot files, „each inside a reported box run"; nothing vanished, nothing unexplained; Tester4 stored its first baseline (44 files); Tester-2's own listing failed (no `snapshots/` dir — the box has never pushed). No `offsite_snapshots_vanished` / `_unexplained` mail in the operator mailbox (second channel). Evidence `audits/due-checks-2026-10-21/R-895-hub-0710.txt`. The positive alarm path is still proven only by the unit red-proofs. | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC |
| **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | NARROWED — **2026-10-23 (D32):** the sheet is current — names and places only, one A4, HU + EN (`runbooks/break-glass-sheet.md`); every place checked to exist. LEFT: the operator prints it, fills it by hand, keeps it outside the home. **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G1/G3/G10):** unchanged; the sheet is still the one act that breaks the circle, and S5's offline form is unknown to CC — on the sheet too (decide list in STATUS). | operator: print the sheet | Operator: print + store the sheet; then close | operator |
| **R-1252** | Backup & restore | P2 | **One rejected backup block in an app's `.felhom.yml` makes that night's second-drive copy DELETE the app's copies of the household's files.** REPRODUCED on 9202 2026-10-24 (`audits/review-round-2-2026-10-24/``proofs/H2-bad-backup-block/`): audiobookshelf with a marker file in `media/audiobooks`; two unchanged runs kept the copy (the positive control); one bogus path in the block → „backup block rejected … matches no compose bind” → the legacy legs (appdata only) → „Tier 2 reconcile: removed stale dest dir userdata/media”; the original on the drive untouched. Mechanism: `internal/stacks/metadata.go:550-557` (any defect rejects the WHOLE block) → `appbackup/classify.go:217-221` → `backup/tier2_capture.go:84-100` → `tier2.go:492``tier2Reconcile` (`:325-329`), against D25's own header (`tier2_kept.go:17-18`). Same family (static): a D25 kept copy is reconciled away when its leg leaves the declared set and its record stays (HUNT-2 H2, `tier2.go:485-492`); Remove with `remove_backups` deletes kept copies and leaves their records (HUNT-2 H9, `api/router.go:1086-1087`). **Fixed on main 2026-10-24 night** (controller `518a135`; Part 2): reconcile only under a valid classification; a rejected block is said on the app's status and once a day to the operator (`backup_run_failures`, leg `tier2-settings-unreadable`); kept copies stay in the keep set (H2); Remove drops the kept records it deleted (H9). **Proven live on 9202** (`audits/restore-holes-2026-10-24/P2-9202/`: two control runs, then the bogus path → „REJECTED … every existing copy … KEPT”). | **VERIFY** — fixed on main, waits for the release | the next controller release | release the controller (Monday list) | CC |
| **R-1253** | Backup & restore | P2 | **An export with „stop the app first” during another app's restore overwrites the ONE app-stop marker file — the restore's crash record and its Undo are lost; a power cut then leaves the restored app on mixed data with no put-back.** Found 2026-10-24 (HUNT-1 H3): `AppStopGuard.Begin` writes unconditionally (`internal/backup/appstop_marker.go:279-292`) and `End()` removes the file (`:306-320`); the exporter takes the guard (`internal/appexport/export.go:139-143,262-266`) and neither `StartExport` (`:173-178`) nor the web handlers (`internal/web/handler_export*.go`, grep: no restore check) look for a running restore. Static. **Fixed on main 2026-10-24 night** (controller `c9b4b82`; Part 3): an export that stops its app refuses (409, „err.appexport.another_operation_running”) while another operation of this process holds the marker (`BusyWith`, `BeginExclusive`); picked over one marker per app (smaller: the exporter and one guard method). **Proven live on 9202** (`audits/restore-holes-2026-10-24/P3-9202/`: refused during a MeTube restore, uptime-kuma never stopped; the same export after it ran). | **VERIFY** — fixed on main, waits for the release | the next controller release | release the controller (Monday list) | CC |
| **R-1254** | Backup & restore | P2 | **The hub saves a household's new escrow package in TWO writes; a failure between them leaves the new repository hash beside the OLD identity package, and the box then marks itself escrowed — on a total loss the written code opens nothing.** Found 2026-10-24 (HUNT-3 H1), read at `file:line`: `hub/internal/api/handler.go:1415``SaveHostEscrow` commits blob + hash, `:1445``SaveHostDRBundle` sets `identity_blob` in a second statement (`internal/store/store.go:3788-3796`; the upsert `:3490-3511` never touches it); the controller's confirm reads the hash (`felhom-controller/controller/internal/report/escrow_confirm.go:195-217`); the agent sends once. Same family (H2): a lost or late answer (30 s) tells the household „the previous code still works” when the hub may already hold the new package (`felhom-agent/internal/localapi/escrow_ceremony.go:79,532-534,639-641`). Static. **The hub half is fixed on main 2026-10-24 night** (felhom.eu `dcf4f84c`; Part 3): key, hash and identity blob in ONE transaction (`SaveHostEscrowWithIdentity`), a bad identity refused before any write; hub image `0.170.0-rc.r1254` built locally (`67db584a45d7`), NOT pushed, NOT deployed. The agent ceremony half (H2: „the previous code still works” after a lost answer) is still open. | **NARROWED** — the hub half fixed on main (waits for the hub release); the ceremony half open | the hub release (attended); the agent half | deploy hub v0.170.0 (Monday list); then the ceremony asks the hub what it holds before it says „the previous code still works” | CC |
| **R-1269** | Backup & restore | P2 | **A reinstall's own capture rewrites the unit that „remove — keep backups” kept, so restoring that unit breaks the app and the household's item is gone.** Catalog sweep 2026-10-24, reproduced on its own (`audits/catalog-sweep-2026-10-24/_unitprobe/tandoor/`): install Tandoor, one account, „Mentés most”, remove keeping data and backups, install again (a volume app: no kept-data question) → 19:50:32 „Recovery unit captured for tandoor” rewrote ONLY the unit's `app.yaml` (checksum `0f366c97…` → `4438c1c5…`, `created_at` 19:45:26 → 19:50:32) beside the OLD data → „Visszaállítás” paired the NEW install's database password with the OLD data: „password authentication failed” ×42 (app) / ×22 (postgres), state `starting`, the account not there. Seen three times (the sweep's Tandoor cycle, this probe, AdventureLog's re-run 20:03:18 → 20:05:32 → restore); AdventureLog's first run worked only because no capture fell between. The comment at `felhom-controller/controller/internal/backup/recovery_unit.go:101-103` says the refresh never rewrites the definition away from its data. Household order: „I reinstalled, my recipes are gone — restore the backup”. The data itself is not deleted (the old volumes are in the unit). **2026-10-25 (D47, controller `02daebe` + `1f9c863`):** every install has an identity, the unit records it, and every unit writer sets another install's unit aside whole (`backups/set-aside/<app>/<stamp>`, listed with „Visszaállítás”). Proven on 9202 (`audits/restore-truth-2026-10-25/after/`): the old unit set aside byte for byte, restored with its own password, the account back. | **VERIFY** — fixed on main, waits for the release | the next controller release | release the controller (Monday list); re-run `tools/unitprobe2.py tandoor` on the released image | CC |
| **R-1270** | Backup & restore | P2 | **A unit restore reports success while the app never came up.** Catalog sweep 2026-10-24: the controller logged „tandoor restored but health check failed: stack tandoor did not reach running state within 1m30s after restore”, and the status the restore page reads said `ok: true` — „A(z) tandoor: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.” (17:32:41; again in the unitprobe, 19:52:38). The household is told it worked (`audits/catalog-sweep-2026-10-24/tandoor/`, `_unitprobe/tandoor/record.json`). **2026-10-25 (controller `b680db8`):** a restore whose app does not reach running is never a success on any path; the household reads that the app has not started yet; the undo copy is kept; a late start turns it into a success. Proven on 9202 (`audits/restore-truth-2026-10-25/after/_pullprobe/`). | **VERIFY** — fixed on main, waits for the release | the next controller release | release the controller (Monday list); D49 asks whether to pour the old data back instead | CC |
| **R-1271** | Backup & restore | P2 | **„Use my kept data” loaded ANOTHER box's database — a future-dated off-site copy of the rehearsal box — under today's kept files.** Catalog sweep 2026-10-24: Tester4's off-site store holds copies from the 2026-10-20 rehearsal, whose box clock was set to 2026-10-28. On the new box the question said „Az adatbázist innen töltjük vissza: távoli mentés, 2026-10-28 07:39” for Paperless-ngx, Nextcloud and Immich, and loaded it: the seeded account / user / album was not there, and Immich never answered again (404 for 20 min). RomM, whose newest copy was its own („saját mentés, 2026-10-24 19:17”), got its item back. `07` §6.5 says the load is refused when the unit „was taken of another drive” — the rehearsal box's drive was another disk, so that check did not hold here (which field it compares: not measured). The household's own local unit stays (nothing deleted). Same future-dated copies as R-1206 (that row: retention only). Evidence `audits/catalog-sweep-2026-10-24/paperless-ngx/`, `nextcloud/`, `immich/`. **2026-10-25 (controller `bd9f4f7`):** cause — the off-site copy's date was the other box's snapshot time, uncapped, and „newer than every local copy” took it; the „another drive” check compares a mount path (the same on every box). Now a copy more than 10 minutes in the future never counts as the newest; alone it is offered, said and not recommended. Immich's 404 was the copy's old subdomain. Proven on 9202 (`audits/restore-truth-2026-10-25/after/_keptprobe/`). | **VERIFY** — fixed on main, waits for the release | the next controller release | release the controller (Monday list); re-run `tools/keptprobe.py` on the released image | CC |
| **R-1254** | Backup & restore | P2 | **The hub saves a household's new escrow package in TWO writes; a failure between them leaves the new repository hash beside the OLD identity package, and the box then marks itself escrowed — on a total loss the written code opens nothing.** Found 2026-10-24 (HUNT-3 H1), read at `file:line`: `hub/internal/api/handler.go:1415``SaveHostEscrow` commits blob + hash, `:1445``SaveHostDRBundle` sets `identity_blob` in a second statement (`internal/store/store.go:3788-3796`; the upsert `:3490-3511` never touches it); the controller's confirm reads the hash (`felhom-controller/controller/internal/report/escrow_confirm.go:195-217`); the agent sends once. Same family (H2): a lost or late answer (30 s) tells the household „the previous code still works” when the hub may already hold the new package (`felhom-agent/internal/localapi/escrow_ceremony.go:79,532-534,639-641`). Static. **The hub half is fixed on main 2026-10-24 night** (felhom.eu `dcf4f84c`; Part 3): key, hash and identity blob in ONE transaction (`SaveHostEscrowWithIdentity`), a bad identity refused before any write; hub image `0.170.0-rc.r1254` built locally (`67db584a45d7`), NOT pushed, NOT deployed. The agent ceremony half (H2: „the previous code still works” after a lost answer) is still open. | **NARROWED** — **the hub half CLOSED 2026-10-25:** hub **0.170.0** deployed on the operator's yes (manifest bump, ArgoCD sync of the hub Deployment only, image digest `6478498d…`; 30 minutes and more with no alarm mail, the mailbox as the control); Tester 1 ran the escrow ceremony end to end through the page's endpoints and the hub stored the new package in one transaction („the new package is on the hub", `audits/release-2026-10-26/escrow-ceremony-tester1.txt`). The ceremony half stays open | the hub release (attended); the agent half | deploy hub v0.170.0 (Monday list); then the ceremony asks the hub what it holds before it says „the previous code still works” | CC |
| **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.****VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3****2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC |
| **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC |
| **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.****What is NOT claimed here:**`restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC |
@@ -185,7 +178,6 @@ stopping line that lies.
| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
| **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
| **R-1278** | Backup & restore | P3 | **A volume app's reinstall does not offer its kept backup.** „Remove — keep my data” removes a volume app's named volumes and keeps only its backup; the kept-data question is about drive folders only (`07` §6.5, decision 36), so the reinstall starts empty and says nothing. Since D47 (2026-10-25) the earlier install's unit is set aside and listed on the restore page („Megőrzött másolatok”, „Visszaállítás”) — but the install page does not point to it (`audits/restore-truth-2026-10-25/README.md`). | OPEN | — | Design: at install, when a set-aside or removed-app unit of the app exists, say so and offer it beside a fresh start; a test | CC |
| **R-1279** | Backup & restore | P3 | **The `.fab` import removes a running app's volumes before its images are pulled — D48 does not cover it.**`felhom-controller/controller/internal/appexport/restore.go` removes the live volumes (`RemoveStackVolumes`, step 1) and pulls at the database start later; a bundle whose version cannot be downloaded then leaves the app gone (the bundle itself stays, so a later import works). Found reading every restore path for D48 (2026-10-25). | OPEN | — | Pull the bundle's images before step 1 (the D48 rule and its two sentences); a test with an unpullable image | CC |
| **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
| **R-164** | Backup & restore | P4 | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:**`ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
| **R-213** | Backup & restore | P4 | **Putting files back in place — the half the recovery screen deliberately does not do.** The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: *a screen that unlocks and then offers to overwrite is two decisions wearing one button*. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. **The operator named its requirement: a live-versus-backup comparison** — the customer must be able to see what would change before anything is overwritten | **DEFERRED** — operator ruling 2026-10-12 (S18, `09` §3 decision 213): designed after the first customers. Stays P4; re-dated: look again when the first paying household has run 30 days. **OPEN — not started, deliberately** | the comparison design (nothing exists for it yet) | After the first customers | Operator + CC |
@@ -223,6 +215,7 @@ stopping line that lies.
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
| **R-1283** | Backup & restore | P3 | **MeTube's downloads are excluded from every backup — make them `optional` (D44 #25).** Catalog `c50adfc` (R-1019) marks the download folders of qBittorrent, Transmission, NZBGet, SABnzbd and MeTube `excluded` (`app-catalog-felhom.eu/templates/metube/.felhom.yml:33`). Risk: a MeTube video may be gone from its source site, so after a drive failure it cannot be fetched again. Pick: MeTube's `media/metube` becomes `optional` — the second-drive copy carries it, the off-site copy does not; NZBGet and SABnzbd stay excluded (re-downloadable). Cost: second-drive space for MeTube's folder. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §7. | OPEN | — | change the class in the catalog template, run the catalog gates (the download-binds gate and its decoys must accept MeTube as `optional`), check on a Tier-0 box that a second-drive copy carries the folder and the off-site run does not | CC |
| **R-1286** | Backup & restore | P4 | **demo-hp's kimai has been HELD since 2026-10-14 09:06Z („held after a failed restore whose rollback also failed") and no row names it.** Seen 2026-10-25 by the D47 upgrade count (attended release run): kimai is the one app of 12 on demo-hp without an install id, because the start-up step skips a held app; `app.yaml` says deployed, no container exists, boot reconciliation leaves it held, the night update leg has skipped it since at least 2026-10-16 (`audits/burndown-apps-2026-10-16/P1/night-chain-log.txt`). A test box; the hold is the designed fail-closed state (R-379), but 11 days with no record is how a held household app would go unseen too. | OPEN | — | read why the 2026-10-14 restore and its rollback failed (demo-hp's controller log of that day, if kept; the hold record); then clear the hold (`--clear-restore-hold kimai` + a restart) or remove the app through the product; check that a held app raises a standing operator signal, else file that | CC |
| **R-1288** | Backup & restore | P4 | **After the household restores an earlier install's set-aside copy, the next backup sets the REINSTALL's own unit aside too — a second kept copy on the restore page and a second operator alarm.** Seen 2026-10-25 on Tester 1 (0.343.0, Tandoor, attended release run): reinstall → its first backup set install A's unit aside (17:31:55) and wrote install B's unit; restoring A's copy makes the app install A again („the restored data belongs to install 795457b5… — its backups follow it (D47)"); the next 20-minute backup step (17:36:54) found B's unit in `primary/` and set it aside. Nothing is lost; the second copy is the fresh reinstall's near-empty data. D47's design does what it says; the question is whether a restore of a set-aside unit should retire the unit it replaces instead of leaving it for the next writer. | OPEN | — | read `07` §8.2d/§8.2e for the intended end state; if the restore should retire B's unit: move it aside AT the restore (one line, no alarm, „replaced by the restored install") with a test; else document the second copy and word its alarm (R-1287) | CC |
## Storage & devices — 14 rows (P3 5, P4 9)
@@ -243,13 +236,12 @@ stopping line that lies.
| **R-1276** | Storage & devices | P4 | **„Remove with data” leaves the app's empty parent folder** (`<drive>/appdata/<app>`) when the template binds a subfolder: Grimmory, RomM, Paperless-ngx (catalog sweep 2026-10-24; 4 KB each; the bound folders were removed). Harmless (empty, not counted as old data), untidy for a household that browses the drive. | OPEN | — | Remove the parent when it is empty after the bound folders go; a test | CC |
| **R-1282** | Storage & devices | P3 | **The „Később" 03:30 media restart is forgotten by a controller restart — store the due time (D44 #24).** The restart is an in-memory timer (`felhom-controller/controller/internal/web/media_events.go``handleMediaRestart`, `time.AfterFunc`); after a controller restart only the notice stays. Risk: media apps do not see the new drives until the household presses „now". Pick: store the due time and run the restart after a controller restart (a due time already past runs at once, within the same rules). Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §11. | OPEN | — | a small controller change + a test that a stored due time survives a restart and fires once; rides a controller release | CC |
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1241** | Security & access | P1 | **2026-10-24: the Felhom half is DONE** — hub 0.168.0 deployed (the tokens sealed; every copy leaving the volume is clean, the live file's old bytes accepted by the operator, R-975 closed), the hub volume out of the Longhorn backup (D40, R-1243 VERIFY), the two old plain copies deleted by the operator. **What stays here is the homelab half:** the open NFS export and Longhorn's login (the homelab session). **A hacked app pod on DooPlex can read the hub's nightly database copies — with every customer's unsealed Cloudflare, tunnel and git token (R-975) — through the Longhorn NFS export, and by a published file-handle escape the operator signing keys on the same disk: whoever breaks one homelab app can take over the boxes' tunnels and sign agent updates.** MEASURED 2026-10-23 (Plex2Shell impact audit, `audits/plex2shell-felhom-impact-2026-10-23/`): export `/mnt/5_hdd/backup/longhorn-pvc` rw, `no_root_squash`, `no_subtree_check`, a sub-directory of sda1, which also holds `/mnt/5_hdd/felhom.eu/felhom-op-*` (no passphrase, R-924); `secure` is void in pods (containerd sets `ip_unprivileged_port_start=0`; a uid-1000 probe with no capabilities bound port 700); the four `felhom-system` volumes have `encrypted=false`; Longhorn's API on :9500 answers any pod with no login (backups, targets, volumes). In the Plex2Shell window neither door was used — verdict NOT REACHED (nfsd accepted 0 new connections 10-10 → 10-23, positive control; no off-schedule Longhorn backup or volume). 2026-10-23 ~15:18Z, operator yes: the export was narrowed from `*` to `10.42.0.0/16` — the home network and the tailnet are closed, pods are not (Longhorn connects from its pod address). Not exercised: the escape itself (Tier-2 machine). | OPEN | the homelab session (DooPlex host + cluster); the operator (R-924) | Homelab: a NetworkPolicy so only `longhorn-system` reaches :2049/:111 and :9500, or the backup store on its own filesystem. Felhom: R-975 sealed in hub 0.168.0 (deploy pending); the hub's plain Longhorn copies → R-1243; a passphrase on the signing keys (R-924) | operator |
| **R-1139** | Security & access | P2 | **Tester 2's break-glass root password was not rotated with the others (R-954, operator decision D1 = A, 2026-10-17): the box has been offline since 2026-10-04.** demo-hp, Tester 1 and demo-felhom were rotated and proven on 2026-10-17 (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | OPEN — filed 2026-10-17 **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a fresh install sets and vaults a new console password).** | Tester 2 reporting again | When it reports: the installer's step 4b alone (`P3/rotate-step4b.sh` shape), then the vaulted value logs in (PVE API 200) and the old one is refused | CC |
| **R-1257** | Security & access | P2 | **D4's root-key-only sshd drop-in reaches BYO hosts too, while `11` §1 says a BYO host gets guest and Docker updates only — never a host change — and the installer's BYO screen lists what it leaves alone.** Found 2026-10-24 (SEC-2 X1), read at `file:line`: `felhom-agent/configs/felhom-os-apply:230` writes the drop-in in every mode (`plan_writes` holds back only policy `oob`, `:1985-2000`; `check_appliance` guards only the OS layer, `:590,1070`); `felhom.eu/scripts/felhom-host-install.sh` (the BYO screen). A BYO owner who logs in to root by password over SSH loses that at the next bundle. Same area: the „in force” check probes only `addr=192.0.2.1`, so a `Match Address <LAN>` block reads as in force (X3, `:2101-2110`); a failing `sshd -t` refuses and undoes the WHOLE bundle (X4, `:2102,2112-2115`). **Operator ruling D43 = A (2026-10-24 evening, by starting the restore-holes run)**: the drop-in applies only in appliance mode. **Fixed on main** (felhom-agent `c8d3c17`; Part 4): policy `appliance`; BYO → nothing written and our copy removed; no proof → left as it is; X3 the in-force check reads every LAN address; X4 a failing `sshd -t` undoes only the SSH step. Bundle harness `ByoKeepsItsSSH` (7 tests, each rule seen failing). | **VERIFY** — ruled D43 = A; fixed on main, waits for the agent release + the operator's vouch | the agent release + vouch (operator) | release the agent, the operator vouches (Monday list); then a signed bundle on a BYO test host shows `removed`/`skipped` | CC |
| **R-1042** | Security & access | P3 | **The boxes' Cloudflare keys cannot change zone settings, so Cloudflare's own „Always Use HTTPS” cannot be switched on.** Measured 2026-10-14 on all three zones (`audits/https-only-2026-10-14/``1A-1-zones-and-keys.txt`): each `cf_api_token` is valid and sees only its own zone, but GET/PATCH `settings/always_use_https`, `ssl`, `security_header` answer 403; the redirect and transform rule phases too. The https-only fix therefore lives on the box (decision 246). A second layer at Cloudflare needs „Zone Settings: Edit” (and, for headers, „Transform Rules”) added to each customer's key — the hub's zone-reach check stays. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | a key permission only the operator can grant | Optional: widen each customer key, then switch „Always Use HTTPS” on per zone | operator |
| **R-1242** | Security & access | P3 | **The demo NAS export `/mnt/5_hdd/felhom-data` → 192.168.0.162 (demo-felhom) is a sub-directory of DooPlex's sda1 with `no_subtree_check`, so a compromised demo-felhom host could walk the whole disk as uid 101000 and read every world-readable file there (the GPG secrets exports are 0644, every repo, the public keys).** Measured 2026-10-23 (`audits/plex2shell-felhom-impact-2026-10-23/` Part 1; `exportfs -v`: `all_squash,anonuid=101000`). 0600 files (the signing keys) stay out of reach. Not exercised. | OPEN | a DooPlex host change (fence) | Give the demo share its own filesystem, or `subtree_check`; or remove other-read from `/mnt/5_hdd/backup/secrets/exports` | operator |
| **R-1104** | Security & access | P3 | **Check https-only on Tester 2 when it reports again.** R-1033 (plain http on every public name) is closed on the three boxes that report (35 names → 301, 2026-10-16). Tester 2 (`sajatfelhom.hu`) has been off since 2026-10-04 on controller 0.292.0; its addresses answer Cloudflare 530 (tunnel down), so nothing is served in plain http while it is off. When it comes back it takes the floor (≥ 0.312.0 carries the redirect). | OPEN — filed 2026-10-16 | Tester 2 reporting again | Run `scripts/https_only_check.py` with Tester 2 in it; close | CC |
@@ -294,15 +286,15 @@ stopping line that lies.
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | OPEN | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC |
| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | OPEN — **now in controller 0.343.0 (released 2026-10-25); the three edges act only on a box whose agent is ≥ 0.165.0 (it alone reports `waiting_for_quiet`) — fix before any household gets agent 0.165.0 by signed update** | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC |
| **R-1262** | Monitoring & notifications | P3 | **The upload watch treats ANY viewer as an explanation for any amount of upload, so a hacked Plex can send freely while one family member watches.** Found 2026-10-24 (SEC-1 X1): `felhom-controller/controller/internal/upwatch/watch.go:275,299`; `08` §9 states the rule, its limits list did not name this case. | OPEN | — | propose a cap (viewers × a bitrate ceiling) for the operator's R-1244 design; `08`'s limits list names the case meanwhile | CC |
| **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC****2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent``acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
| **R-1250** | Monitoring & notifications | P3 | **A qBittorrent that seeds all day is never line-tested, so it never gets the limit that exists for it.** The weekly test (R-1247) starts only after 10 minutes in which the host's uplink sent < 2 Mbit/s each minute; a seed at the line's speed keeps it busy through every window, so no test, no tested line, and no qBittorrent limit (D41 uses the tested line only). Found 2026-10-24 by reading the rule against the case it serves; not yet seen on a box (no test box seeds). | **NARROWED 2026-10-24 — FIXED on main, not released** (brief 2026-10-24 Part 4, commit only): agent `29bcf44`+`1e087bb` (`waiting_for_quiet`, `POST /host/line-test/run`), controller `046b808` (`internal/upwatch/linehold.go`: record first, hold at 1 KiB/s, test, the exact old limit back; the start-up check after a kill; the 50 % rule waits). Red-proofs in both CHANGELOGs. **Live on demo-hp with a real seed** (10 Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; held: uplink 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, back to 1 249 280 B/s exactly; a kill -9 mid-test left the record and the start-up check put the limit back (`audits/dooplex-alarms-2026-10-24/part4-r1250-live-proof.txt`). The live run used probes over plain HTTP (the pinned agent client is unit-tested, not live). | the next release train (agent ≥ 0.165.0 needs the operator's vouch) | Release agent + controller in the next normal train; on a seeding box read one Tuesday's `line test on request` line; then close | CC |
| **R-1250** | Monitoring & notifications | P3 | **A qBittorrent that seeds all day is never line-tested, so it never gets the limit that exists for it.** The weekly test (R-1247) starts only after 10 minutes in which the host's uplink sent < 2 Mbit/s each minute; a seed at the line's speed keeps it busy through every window, so no test, no tested line, and no qBittorrent limit (D41 uses the tested line only). Found 2026-10-24 by reading the rule against the case it serves; not yet seen on a box (no test box seeds). | **VERIFY — released 2026-10-25, the live proof is a dated check:** controller 0.343.0 (global floor 17:53Z) + agent 0.165.0 (vouched 18:38 CET) carry it; the real proof cannot run before the next due window — the agent starts a test only on its due Tuesday 10:00–15:59 local, and every box measured on 2026-10-24. **DUE 2026-10-27:** a throwaway qBittorrent on demo-hp seeding to DooPlex's qBittorrent (operator's yes 2026-10-25, about 30 minutes), from before the host's slot; read the agent's line value and the limit back. **Earlier: NARROWED 2026-10-24 — FIXED on main, not released** (brief 2026-10-24 Part 4, commit only): agent `29bcf44`+`1e087bb` (`waiting_for_quiet`, `POST /host/line-test/run`), controller `046b808` (`internal/upwatch/linehold.go`: record first, hold at 1 KiB/s, test, the exact old limit back; the start-up check after a kill; the 50 % rule waits). Red-proofs in both CHANGELOGs. **Live on demo-hp with a real seed** (10 Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; held: uplink 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, back to 1 249 280 B/s exactly; a kill -9 mid-test left the record and the start-up check put the limit back (`audits/dooplex-alarms-2026-10-24/part4-r1250-live-proof.txt`). The live run used probes over plain HTTP (the pinned agent client is unit-tested, not live). | the next release train (agent ≥ 0.165.0 needs the operator's vouch) | Release agent + controller in the next normal train; on a seeding box read one Tuesday's `line test on request` line; then close | CC |
| **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Design note (2026-10-21, after R-1192's diagnosis): the freeze's signature is cheap to see from the HOST side — the guest cgroup's memory PSI „full" above ~50 % for a minute, or its `memory.events max` climbing, while the host still reports; the agent could raise `guest_memory_thrash` (operator mail in its own words, not „no report for 45 m") and the household banner „A dobozod most nagyon lassú…". With the swap fix (R-1192) the measured thrash ends within a minute, so this alarm should be built only if a freeze is seen again after the agent release; until then it stays a design. Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC |
| **R-1202** | Monitoring & notifications | P3 | **A box whose clock is behind blames the hub and stops an app without telling anyone.** Rehearsal 2026-10-20 B6 #8: RTC set 2 years back (the OS raised it to 2026-04-13): the dashboard shows only „Hub nem elérhető" while the log says „x509: certificate … not yet valid: current time 2026-04-13…"; Paperless crash-looped and the box stopped it with a restore hold; the `app_stopped_unhealthy` event failed (hub unreachable), so neither the household nor the operator heard. After the real time came back the app stayed stopped until a Start (clear sentence, 1 min). | **OPEN** | — | say „the box's clock is wrong (it says <date>)" when TLS fails with not-yet-valid/expired against a hub that answers; keep the stop event queued until the hub is reachable | CC |
| **R-340** | Monitoring & notifications | P3 | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice****REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc/<pid>/fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC |
@@ -320,9 +312,10 @@ stopping line that lies.
| **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC |
| **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
| **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
| **R-1287** | Monitoring & notifications | P3 | **A D47 set-aside mails the operator as „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" — no backup failed; the body says the right thing („its backup belonged to an earlier install … SET ASIDE whole").** Seen 2026-10-25 on Tester 1 with controller 0.343.0 (attended release run): two such mails at 17:31:56Z and 17:36:55Z. The controller sends the set-aside as `backup_run_failures` leg `unit-set-aside`, `Failed: 1` (so the hub never drops a line that claims zero); the hub's subject for that event is fixed („N of M apps failed to back up", `hub/internal/notify/templates.go:353`). The CLAUDE.md corollary: an alarm's text must say what it counts. After the floor, every household reinstall mails the operator a red „failed to back up". | OPEN | — | two-repo: the hub words a `unit-set-aside` leg on its own („… a backup of an earlier install was set aside — nothing failed", warning, not error) with a test on the rendered subject; or a new event type through the allow-list (`hub-event-allowlist-gotcha`); one hub release | CC |
## Hub & operator — 12 rows (P3 5, P4 7)
## Hub & operator — 13 rows (P3 5, P4 8)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -338,6 +331,7 @@ stopping line that lies.
| **R-1002** | Hub & operator | P4 | **A box the household switched off reads ONLINE on the hub's hosts page for its first 45 minutes** (seen live 2026-10-13, R-998 proof, `audits/poweroff-2026-10-13/B-hub-watch.log`: the „Power: Kikapcsolva…" line at 18:53, the KIKAPCSOLVA badge only at 19:38, when the host report turned stale). The customer page read „warn" in part of that gap (seen, not captured). No alarm and no mail fired; the operator page is only slow to say it. | **READY** — filed 2026-10-13 | — | CC: let `hostDisplayStatus` read `deliberate_off_at` before staleness, with a render test per state; next hub release | CC |
| **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator |
| **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator |
| **R-1289** | Hub & operator | P4 | **The hub's „newest controller the fleet reported" can be a dev build that no box runs: `semver.Compare` returns 0 (equal) for any version with a suffix, so „0.343.0-dev.r1251a" — once read first — beats nothing and is never beaten.** Seen 2026-10-25 in the vouch line (attended release run): „golden 0.331.0 unchanged (behind the fleet's 0.343.0-dev.r1251a …)" while the four hosts run 0.343.0 / 0.342.0. `store.NewestReportedControllerVersion` scans every report ever stored (`hub/internal/store/store.go:2210`), and `hub/internal/semver/semver.go:14` gives up on `0-dev…` / `0-rc.N`. Used by the R-120 golden gate (`hub/internal/web/configs.go:1548,1554`): with a suffixed „newest" the gate compares against whatever row came first — today it only words a log line and fails open. | OPEN | — | skip suffixed versions in `NewestReportedControllerVersion` (or compare with the pre-release rule the controller uses, `util.ParseVersionPre`) + a test with a dev and an rc row ahead of a release row (red first); one hub release | CC |
## Business & legal — 14 rows (P2 4, P3 2, P4 8)
@@ -359,7 +353,7 @@ stopping line that lies.
| **R-990** | Business & legal | P4 | **Operator: read `documentation/audits/forum-research-2026-10-12/FELHOM.md` §5 and rule on each recommendation.** READ-ONLY research 2026-10-12: three years (#7943–#11092, 2,908 posts, 195 people, 82 beginner threads) of the public PROHARDVER! topic „Home server / házi szerver építése”, tagged and counted by person (`FINDINGS.md`, method and spot-check in `METHOD.md`; no username or post text in git). Ten ranked recommendations, none acted on; the top three: a „which machine works” page (8 of 33 beginners who name their hardware own only a Synology, a router disk or a Raspberry Pi — the installer cannot run there; the same gap is R-989's first recommendation from a different source, and the shipped-page half is R-991), „you do not choose an operating system — we do” (the biggest beginner question, 20 of 82 threads), and „bring your old office PC” as the cheap entry. Catalog gap (≥ 3 people): a DLNA server (7), an ad/DNS blocker (5), a camera recorder (5). | **WAITING-ON-OPERATOR** — filed 2026-10-12 | the operator's reading | Operator: rule on each of the ten (yes / no / later). If nothing is done: nothing changes; the research ages | operator |
| **R-1035** | Business & legal | P4 | **Did the „felhő tárhely” page earn anything? Two dated checks.****Baseline (reported by the operator from Search Console, 2026-10-14, last 3 months):** 1 click, 48 impressions, CTR 2.1 %, average position **73.6**; „felhő tárhely” 19 impressions, „felhőtárhely” 5, „felhő alapú tárhely” 3. CC measured separately that the phrase „felhő tárhely” appeared **0 times** on the home, FAQ and backup pages — Google was showing the home page for those queries by accident. `/sajat-felho-tarhely` + `/en/own-cloud-storage` published **2026-10-14**; full baseline, price sources and the power provenance in `audits/seo-2026-10-14/BASELINE.md`. **What to measure, both times:** impressions and average position for „felhő tárhely” and „saját felhő tárhely”, plus the new page's own impressions, clicks and position. **A null result at +4 weeks means nothing on its own** — a new page on a site at position 73.6 takes weeks to months to move, and „felhő tárhely” alone is held by Microsoft, Google and the big list sites; the page aims at the two longer phrases. That is why there are two checks and not one, and why neither should be read as a verdict on the copy. **There is no Search Console connector in this project, so CC cannot read these numbers** — the operator reads them, CC records and compares. **Blocked until the operator asks for indexing** (Search Console → URL-ellenőrzés → the URL → „Indexelés kérése”): until that happens, or Google finds the page itself, both checks read zero for a reason that is not the page. **This is a slow channel and does not replace talking to people.** | **WATCHING** | operator: request indexing, then supply the Search Console read | 2026-11-11: first read (+4 weeks); 2026-12-09: second (+8 weeks), compared with the first | operator + CC |
## Process & tooling — 32 rows (P3 7, P4 25)
## Process & tooling — 33 rows (P3 8, P4 25)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -395,6 +389,7 @@ stopping line that lies.
| **R-1236** | Process & tooling | P3 | **Gitea (the code and the registry) is OOM-killed at its 1 GiB limit — 4 restarts in 17 days, the last at 06:41:10 UTC on 2026-10-23, six seconds after the global floor moved three boxes to a new controller image.** Read 2026-10-23 (`kubectl get pod`: `lastState.terminated.reason OOMKilled`, exit 137; kernel: „git invoked oom-killer”; limit `memory: 1Gi`, request 256Mi; 620 Mi in use at rest). During a restart a box update, an agent package fetch, a CI clone or a push fails. Not changed: DooPlex's configuration is the operator's. | OPEN | the operator's yes (a DooPlex change) | Operator: raise the limit (e.g. 2 GiB; DooPlex has the RAM) — CC's pick; or keep it and accept a restart in release hours | operator |
| **R-1264** | Process & tooling | P4 | **Skills tooling (review round 2, DOC-B U7/S22):** the grandfathered skill has no size cap — felhom-build-deploy is 240 lines while R-394 recorded 179 (`scripts/check_skills.py:45-47,98-101`); a NEW skill folder (`skills/felhom-web-design`) is not live until `install_skills.py` runs again (not run: no DooPlex change in this unattended run). | OPEN | — | cap the grandfathered entry at its current size; run install_skills.py in an attended session | CC |
| **R-1277** | Process & tooling | P4 | **The catalog's box fixtures drifted from the apps they test** — fixed only in the sweep's copy (`audits/catalog-sweep-2026-10-24/tools/upgrade_fixtures_box.py`, marked SWEEP COPY): OpenGist 1.15 signs up at `/-/register`; Calibre-Web's admin is the generated ADMIN_USER; Mealie's after_install replaces the factory password (the seed raced it); Papra 26.6 needs an Origin header; Rallly 4.15 closed password sign-up (no fixture possible by password); Vaultwarden 1.37.4 answers 404 on `POST /identity/accounts/register` (the fixture was written on 1.36; the admin invite works, the invited sign-up has no route — the newer clients' register/finish flow);the read-back through a family gate needs a fresh cookie after the setup gate opens (Grimmory stayed inconclusive). New fixtures for Homebox, Immich, Komga, Jellyfin, Ghost, Radarr, Sonarr in `tools/sweep_fixtures.py`; `box_walk.py` gained a `sweep` target. | OPEN | — | Port them into `app-catalog-felhom.eu/scripts/` with the bench tests | CC |
| **R-1290** | Process & tooling | P3 | **The agent's CI is red since 2026-10-25 ~18:00Z: the registry no longer serves agent 0.159.0, 0.159.1, 0.159.2 and 0.160.0, while the `published` gate asserts the newest 10 released versions are downloadable (`felhom-agent/scripts/retention-policy.json`, „what the registry keeps").** Measured 2026-10-25 (attended release run): CI job 2326 (`4697f8a`, after the 0.165.0 upload) green; job 2334 (`6b6ba0c`, docs only) red on those four (HTTP 404); the registry lists 7 versions — 0.158.1, 0.161.0–0.165.0 and a test build `0.163.0-step1`. Our scripts delete only the version they re-upload (`publish-agent.sh:77`), so the prune is outside the repos (a Gitea package clean-up rule or a hand prune on DooPlex — not CC's to change). No box needs those versions (all run 0.165.0; MinAgent 0.158.1 is served), but every agent push now mails a CI failure and hides a real one. | OPEN | the operator (DooPlex's Gitea) | the operator: name what pruned on 2026-10-25 and its keep rule; then CC sets `generic_versions_kept` to that rule (or bounds the gate on the hub's MinAgent, R-287's follow-up) in one commit, and CI is green again by hash | operator |
<!-- DUE-CHECKS-BEGIN — machine-readable. Parsed by scripts/due_checks_gate.py.
One row per dated check. The R-number must have a row above. Dates are UTC.
@@ -406,6 +401,7 @@ stopping line that lies.
|---|---|---|
| R-435 | 2026-10-27 | Tester 1's weekly off-site clean-up ran inside a hub window with no drop alarm (box log + mailbox); detail in the R-435 row |
| R-1183 | 2026-10-26 | every hub daily job ran ONCE on 2026-10-25 (hub log + one snapshot file per slot — four slots since hub 0.167.0, D34, the 02:00 one not twice); detail in the R-1183 row |
| R-1250 | 2026-10-27 | the released line hold, live: a seeding qBittorrent on demo-hp (peer DooPlex's, operator yes 2026-10-25) through the host's due window; the line value and the limit back; detail in the R-1250 row |
| R-1228 | 2026-10-30 | D35 (yes, after the release week): start moving the website + installer to Cloudflare; detail in the R-1228 row |
| R-1174 | 2026-10-27 | the status gate refuses from today: one conviction seen exiting 1 (a hand run on a copy of the page is enough); detail in the R-1174 row |
| R-1050 | 2026-11-15 | HSTS one year if no https trouble since 2026-10-15; detail in the R-1050 row |
@@ -7,6 +7,13 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
## Moved from STATUS on 2026-10-25 (attended release run)
**Updated 2026-10-25 (Sunday).** A reinstall keeps old backups whole; a restore downloads first and tells the truth.
Decided the same day: **D49 = A**, **D46 = yes** (`09` §3 decisions 258, 261):
| D49 | R-1270 | A restore's data is back but the app does not start in 90 s. Keep the restored data and wait, or pour the old data back by itself? | **Keep and wait** (a slow first start is common; the old copy is kept, a late start turns it into a success; you get one alarm) | pour back by itself: a slow but correct restore is thrown away | the data stays restored, the old copy stays for support | — |
| D46 | R-1272 | Put Homepage behind the family gate, like Glance? | **Yes** (a stranger now sees the box's CPU, memory and disk, and later the family's links; one line in the template, tried on the drill first) | leave it open: anyone who finds the address sees those | it stays open to strangers | — |
The Monday list, run on 2026-10-25 (attended; results in `REPORT-release-2026-10-26.md`):
**Monday 2026-10-26, release list (attended).** 1. Controller v0.343.0: the restore and backup fixes (R-1251–R-1253, R-1269–R-1271), R-1250. If not: the P1 stays open. 2. Agent v0.165.0 + your vouch: BYO keeps its SSH (R-1257). 3. Hub v0.170.0: the recovery package in one step (R-1254). 4. Catalog: n8n usage data off; OpenGist 256 MB.
## v0.170.0 (2026-10-25) — the escrow package in one transaction (R-1254) — built for release in the attended release run; deployed only on the operator's yes
## v0.170.0 (2026-10-25) — the escrow package in one transaction (R-1254) — DEPLOYED on the operator's yes 2026-10-25 15:50 UTC (image `sha256:6478498d…`; ArgoCD sync of the hub Deployment only); Tester 1's escrow ceremony stored its package in one transaction; no alarm mail in the watch after
- **The new escrow key, its sealed-password hash and the identity blob are stored in ONE transaction**
(`Store.SaveHostEscrowWithIdentity`; `SaveHostEscrow` is it with no identity blob). Until now the escrow PUT committed
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
admin committed