R-1295 live on hub rc.1 (a fresh Tester4 install set up its whole-guest off-site tier; the box consumed the credential and made its first copy); R-1294 read: an honest plan the cap can never catch up with (D54); R-1291 design (08 §10, D53); design lines for R-1295, R-1285, R-1292, R-1294; R-1296 filed; STATUS.

documentation/backlog/OPEN-ITEMS.md

@@ -145,7 +145,7 @@ stopping line that lies.
 | **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | WATCHING — **2026-10-22 night:** not reproducible tonight; the Immich server log of 19:38 is gone (Immich restarted at 02:16 for the off-site dump). What the household sees: the upload fails after 10 minutes and the Immich phone app sends it again. | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC |
 | **R-1267** | Apps & catalog | P4 | **Every recreate of Immich leaves one empty anonymous Docker volume behind: the image declares `VOLUME /data`, the template does not map it.** Found 2026-10-24 (review round 2, while putting 9202 back): 9202 holds 296 empty anonymous volumes; each night chain's recreate of `immich-server` made a new one (12:58, 13:00, 13:01 — the three test runs; the two dangling ones this session made were removed by name, nothing pruned). The photos are NOT affected: Immich picks `/usr/src/app/upload` whenever both folders exist (`detectMediaLocation` in the image's `server/dist/services/storage.service.js:28-39`), and the template binds that folder (`templates/immich/docker-compose.yml:49`) — NOT-A-FINDING for data placement. **Seen again 2026-10-24 night on Tester 1:** privatebin's image declares `/run`, `/tmp` and `/var/lib/nginx/tmp` as volumes — each recreate (the night dump's stop/start) leaves three empty anonymous volumes; the same class as Immich's `/data` (`audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt`). | OPEN **2026-10-24 (catalog sweep):** the same class for OnlyOffice and RomM (RomM's image volume `/romm` holds only the empty mount points of its binds); after every catalog app had been removed WITH data, the sweep box still held 52 such volumes (PrivateBin 21, OnlyOffice 8, RomM 8, Immich 3, and 12 from 12:50–12:52 before the batch, holding the same nginx folders as PrivateBin's) — remove-with-data does not see them (no compose project label). All small (4–72 KB), no household data (`audits/catalog-sweep-2026-10-24/_box/anon-volumes.txt`). | — | declare the image's VOLUME paths as tmpfs or named volumes in the catalog (Immich, privatebin), or remove the dangling ones a recreate leaves, by name | CC |
 | **R-1268** | Backup & restore | P3 | **Remove with „remove backups” leaves a restore's undo copy on the drive (`backups/restore-undo/<app>/`).** Seen 2026-10-24 night on Tester 1 (restore-holes Part 1, run 1): MeTube was held for support after a failed crash put-back, then removed with `remove_backups=true` — the 1.5 GB copy stayed (the removal deletes `backups/primary` and the second-drive mirror, not `restore-undo`), and nothing lists it (`audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt`; removed by hand, CC's own test data). The copy is kept on purpose while a hold exists; after the app is gone it is orphaned. | OPEN | — | Remove with `remove_backups` also deletes the app's `restore-undo` folder (under the same path guard), or lists it as a kept copy | CC |
-| **R-1292** | Apps & catalog | P4 | **qBittorrent's 50 % rule reads and writes the limit of the CURRENT speed mode: with the household's alternative speed on, it sets the alternative limit, and a later switch reads as „the household changed it" — the rule stops for good.** Measured 2026-10-25 night on qBittorrent 5.2.4 (`transfer/uploadLimit`/`setUploadLimit` follow the mode; `audits/night-2026-10-25/r1261-qbittorrent-api-measured.txt`); found while fixing R-1261 (the hold now names its slot). | OPEN | — | make the rule own the NORMAL slot only (`up_limit` through `app/preferences`, as the hold does) + a test with the alternative mode on (red first); one controller release | CC |
+| **R-1292** | Apps & catalog | P4 | **qBittorrent's 50 % rule reads and writes the limit of the CURRENT speed mode: with the household's alternative speed on, it sets the alternative limit, and a later switch reads as „the household changed it" — the rule stops for good.** Measured 2026-10-25 night on qBittorrent 5.2.4 (`transfer/uploadLimit`/`setUploadLimit` follow the mode; `audits/night-2026-10-25/r1261-qbittorrent-api-measured.txt`); found while fixing R-1261 (the hold now names its slot). | **VERIFY** — 2026-10-26: controller `2619191` (unreleased) — the 50 % rule owns the NORMAL slot only (`up_limit` via app/preferences); test with the alternative mode on, red-proved; live on Tester 1 (rc `0.345.0-rc.1`; 9202 has no tested line, so the rule never acts there): `audits/r1292-qb-normal-slot-2026-10-26/`. **Earlier:** OPEN | — | make the rule own the NORMAL slot only (`up_limit` through `app/preferences`, as the hold does) + a test with the alternative mode on (red first); one controller release | CC |
 
 
 ## App updates — 5 rows (P2 1, P3 4)
@@ -212,8 +212,8 @@ stopping line that lies.
 | **R-1232** | Backup & restore | P2 | **A reinstalled box may take its first whole-box copy to ep0 up to a week late: the old box's copy in the same household space seems to count as its last one.** Test night 2026-10-23 (`audits/night-2026-10-23/P4-testnight/`): the new Tester4 box (guest 9201) had its PBS tier applied at 19:28 UTC, but no whole-box run to ep0 came all night; the only archive in the household's ep0 space is the rehearsal box's `ct/9201` of 2026-10-20 (same guest number), and the tier's cadence is 168 h. The agent's due-check reads the newest archive for that vmid on the tier's storage (`localapi/server.go` `newestArchiveOn`) — so the old box's copy reads as „2.6 days old, not due”. If so, the ep0 copy is the OLD box's for up to a week. PLAUSIBLE — read from the code and the night's silence, not proven by a log line (the controller logs the due-check only at debug). | OPEN | — | Prove: read `GET /backup/tiers` for the box (the agent's answer per tier) on the next reinstall; fix: a tier's „last backup” counts only archives this host wrote (or after the host's enrollment); an agent test with a foreign archive in the namespace | CC |
 | **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
 | **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
-| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
-| **R-1295** | Backup & restore | P3 | **A re-enrolled customer's whole-guest off-site tier (PBS DR) can fail to provision at the fresh install and only the hub log says so.** Seen 2026-10-26 on Tester4's fresh install (golden 0.344.0 proof, host `Tester4-eb7371`): 10:31:15 CET `[ERROR] pbsdr auto-provision for Tester4 (WG-registration hook): pbsdr: F-14 auto re-issue for Tester4: tenantsync: endpoint refused: own-namespace list as felhom@pbs!Tester4 failed twice — rolled back (code self_check_failed) — save the customer config to retry`. No operator mail came (mailbox read 09:33Z: only the household's setup-code mail). On a household reinstall this would leave the box without its whole-guest off-site copy until someone reads the log. Not retried here: the retry lists ep0, fenced in this run. Evidence `audits/release-monday-2026-10-26/fresh-install/proof.txt`. | OPEN | — | read why the own-namespace list fails right after the re-issue (a token not yet valid on ep0? a namespace left by the deleted host?); make the rollback mail the operator (an event through the allow-list); retry once with the operator present | CC |
+| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **WAITING-ON-OPERATOR** — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
+| **R-1295** | Backup & restore | P3 | **A re-enrolled customer's whole-guest off-site tier (PBS DR) can fail to provision at the fresh install and only the hub log says so.** Seen 2026-10-26 on Tester4's fresh install (golden 0.344.0 proof, host `Tester4-eb7371`): 10:31:15 CET `[ERROR] pbsdr auto-provision for Tester4 (WG-registration hook): pbsdr: F-14 auto re-issue for Tester4: tenantsync: endpoint refused: own-namespace list as felhom@pbs!Tester4 failed twice — rolled back (code self_check_failed) — save the customer config to retry`. No operator mail came (mailbox read 09:33Z: only the household's setup-code mail). On a household reinstall this would leave the box without its whole-guest off-site copy until someone reads the log. Not retried here: the retry lists ep0, fenced in this run. Evidence `audits/release-monday-2026-10-26/fresh-install/proof.txt`. | **VERIFY** — 2026-10-26 afternoon: **cause measured** — PBS's proxy caches permissions up to 5 s and an ACL-only change does not clear it; under concurrent requests the new token is refused for up to ~5 s and the script's one quick retry falls in the window (throwaway PBS 4.2.5 under load: v1.3.0 11/30 failed, v1.4.0 0/30; candidates ruled out with evidence, `audits/r1295-pbsdr-selfcheck-2026-10-26/`). Fixed: tenantsync v1.4.0 (`722013da`, waits up to 15 s, the admin token too) — **not installed on ep0** (an ep0 change, the operator's yes); hub v0.172.0 (`722013da`) retries a rollback 3×, 7 s apart, and a final failure mails the operator (`pbsdr_provision_failed`, operator-only). Red-proved. **Live on hub rc.1:** a fresh Tester4 install provisioned on the first try; the box consumed the credential 12:07:25Z and made its first whole-guest copy (1 PBS snapshot). Left: the hub release and the ep0 script install. **Earlier:** OPEN | — | read why the own-namespace list fails right after the re-issue (a token not yet valid on ep0? a namespace left by the deleted host?); make the rollback mail the operator (an event through the allow-list); retry once with the operator present | CC |
 
 ## Storage & devices — 13 rows (P3 4, P4 9)
 
@@ -278,10 +278,10 @@ stopping line that lies.
 | **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
 | **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
 | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
-| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
+| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
 
 
-## Monitoring & notifications — 23 rows (P2 1, P3 16, P4 6)
+## Monitoring & notifications — 24 rows (P2 1, P3 16, P4 7)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -307,8 +307,8 @@ stopping line that lies.
 | **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC |
 | **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
 | **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
-| **R-1291** | Monitoring & notifications | P3 | **A held app gives the operator no lasting signal: one event at the stop, then nothing — demo-hp's Kimai sat held 11 days unseen.** Found 2026-10-25 night (R-1286). No held-app state reaches the hub (the report carries none; the hub shows none); the crash-loop stop sends `app_stopped_unhealthy` once. On a household box an app held after a crash loop, a failed update or a failed restore stays stopped until someone opens its page. | OPEN | — | a two-repo design: the controller reports its holds (stack, reason, since), the hub shows them on the host and mails a reminder after N days (the D-level of the reason); one row's worth of design first | CC |
-
+| **R-1291** | Monitoring & notifications | P3 | **A held app gives the operator no lasting signal: one event at the stop, then nothing — demo-hp's Kimai sat held 11 days unseen.** Found 2026-10-25 night (R-1286). No held-app state reaches the hub (the report carries none; the hub shows none); the crash-loop stop sends `app_stopped_unhealthy` once. On a household box an app held after a crash loop, a failed update or a failed restore stays stopped until someone opens its page. | **WAITING-ON-OPERATOR** — 2026-10-26: the design is written (`08` §10 [DESIGN — proposed]: `app_held` / `app_hold_cleared` events + a daily re-send, a „Held apps" list on the host page, an operator reminder after a grace per reason, the hub never clears a hold; ~½ day controller + ~1 day hub). The pick is STATUS D53. **Earlier:** OPEN | — | a two-repo design: the controller reports its holds (stack, reason, since), the hub shows them on the host and mails a reminder after N days (the D-level of the reason); one row's worth of design first | CC |
+| **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC |
 
 ## Hub & operator — 12 rows (P3 3, P4 9)