R-1299 CLOSED: the clean-up window button (hub 0.174.0/0.174.1, live grant + cancel on tester-1); R-1326 filed (hub start-up vs liveness); R-1237 updated (46 dead names removed with sec rm); 07/05 design lines; secrets runbook: sec rm

documentation/backlog/CLOSED-ITEMS.md

@@ -26,6 +26,12 @@
 
 ---
 
+## 2026-10-27 (evening) — R-1299 closed: the clean-up window button (full text of the row: `git show 714ee236:documentation/backlog/OPEN-ITEMS.md`)
+
+| Row | What | Closed | Evidence |
+|---|---|---|---|
+| **R-1299** | **The operator's one-shot clean-up grant (`POST /offsite/window-grant/<customer>`, optional `max_remove`) has no button on any hub page** (P4) | CLOSED 2026-10-27 evening — hub v0.174.0 (`d85198b8`, CI 2433) + v0.174.1 (`de57c090`, CI 2437; the event row and the Backup & DR tab). The customer page's „Backup & DR" tab has the „Off-site clean-up window" card: state, an optional limit (refused above the store's snapshot count; ceiling 500), a confirm step, the same route and rule; a waiting grant can be cancelled; every grant and cancel is a stored operator-only info event. Live: the operator granted on tester-1 (20:07 CET, grant read back `1`, log „by operator session") and cancelled on 0.174.1 (20:42 CET: grant gone, one stored cancel row). Nothing was removed. | `audits/r1299-window-button-2026-10-27/` |
+
 ## 2026-10-27 (day, attended) — security app updates: three rows closed; R-1314 was filed and closed the same day (full text of the rows: `git show 708a3451:documentation/backlog/OPEN-ITEMS.md`)
 
 | Row | What | Closed | Evidence |

documentation/backlog/OPEN-ITEMS.md

@@ -268,7 +268,7 @@ stopping line that lies.
 | **R-1050** | Security & access | P4 | **Raise HSTS to one year on or after 2026-11-15, if no https trouble was reported.** `09` §3 decision 249 (D2 = A, operator 2026-10-15): HSTS stays `max-age=86400` now; after a month with no https trouble it becomes one year (`31536000`); never preload, never `includeSubDomains`. The value is `HSTSMaxAgeSeconds` in `felhom-controller/controller/internal/infra/infra.go`, pinned by `TestRenderForwardedHeaders_HTTPSOnly`. „Trouble” = any report of a name that will not open, or a box answering http after the floor. | **WATCHING** — filed 2026-10-15 | the date | On or after 2026-11-15: read the register and the operator's mail for https trouble since 2026-10-15; none → a controller release with one year (test updated), bake → vouch → floor | CC |
 | **R-1054** | Security & access | P3 | **The hub's operator/global API key is still one bearer that can do everything on `/api/v1`, including reading every box's break-glass root password (`GET /admin/hosts/{id}/recovery-credential`).** Hub v0.152.0 gave the web surface scoped tokens (`05` §17) but tokens are refused on `/api/v1` by design. Who uses the global key today is not listed in one place. Fix: list its consumers; move each operator use to a scoped token on the web side (or a token class on the API), and keep the global key only where a box path needs it — or retire it. | OPEN — filed 2026-10-15 by the token build | — | List the global key's consumers (pod env, scripts, runbooks) | CC |
 | **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f <conf>` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC |
-| **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC |
+| **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | **OPEN** — 2026-10-27 (operator yes): the dead names were REMOVED from the store with `sec rm` (46, one commit each; `audits/r1299-window-button-2026-10-27/sec-rm.txt`, names only): every one-time code this row names is gone; R_DEMO_FELHOM stays (unsure). Consequence: the nightly sweep no longer watches for the removed values — they open nothing (their boxes are destroyed). The session transcripts themselves are unchanged — that question stays here. **Earlier:** OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC |
 | **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator |
 | **R-1309** | Security & access | P4 | **What no automated source shows: a fix announced only by mail, forum or a blog we do not read; a security fix nobody calls one (cloudflared 2026.10.2's „bump distroless image"); closed-source apps (Plex, Emby).** The watch (R-1303) shows these only as BEHIND (weekly). The vendor mails the operator receives stay a source. `audits/security-watch-2026-10-26/SPIKE.md` | OPEN — filed 2026-10-26 (a stated limit) | — | none until a vendor offers a feed; add a vendor's feed to `security_watch.FEEDS` when one exists | CC |
 | **R-1312** | Backup & restore | P3 | **The hub customer page's off-site switch did not reach the box: on Tester 1 (2026-10-27, operator-attended, ~14:00–14:12 UTC) the box's saved off-site target stayed `enabled` and its `controller.yaml` `offsite.enabled: true` (unchanged since 2026-10-04) while the hub showed off-site OFF.** So what the switch stops on the box side is not known: the controller reads `cfg.Offsite.Enabled` only at start (`cmd/controller/main.go:890`), and nothing rewrote the file. Either the switch is hub-side only by design (then the page must say so), or a box keeps pushing off-site while the operator believes it is off. Found by the R-1218 live attempt (R-1218 closed 2026-10-27 with its live proof waived by the operator). | OPEN | — | Read the hub handler behind the switch and what it changes (config, tenancy, storage access); then one attended run: switch off, watch the box's next off-site push (does it run, does it fail at the storage) | CC |
@@ -319,7 +319,7 @@ stopping line that lies.
 | **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
 | **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC |
 
-## Hub & operator — 12 rows (P3 3, P4 9)
+## Hub & operator — 13 rows (P3 4, P4 9)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -335,6 +335,7 @@ stopping line that lies.
 | **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator |
 | **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator |
 | **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
+| **R-1326** | Hub & operator | P3 | **The hub's start-up eats half its liveness margin: hub 0.174.0 took 36 s from „starting" to „Listening on :8080" (0.174.1: 31 s), and the liveness probe (`initialDelaySeconds` 5, `periodSeconds` 30, `failureThreshold` 3) kills the container after ~65 s of failed probes.** Seen 2026-10-27 at both deploys: two `Liveness probe failed … connection refused` events at start, then ready. The hub's database keeps growing (559 MB on 2026-10-27 04:05); a start-up past ~65 s would restart the hub in a loop at every deploy. No startup probe is set. | OPEN | — | Measure what the start spends its 30 s on (DB open, migrations, checkers); then a `startupProbe` (e.g. 10 s × 18) in `manifests/hub.yaml`, or move the slow work after `Listen` | CC |
 
 ## Business & legal — 14 rows (P2 4, P3 2, P4 8)
 
@@ -355,7 +356,7 @@ stopping line that lies.
 | **R-990** | Business & legal | P4 | **Operator: read `documentation/audits/forum-research-2026-10-12/FELHOM.md` §5 and rule on each recommendation.** READ-ONLY research 2026-10-12: three years (#7943–#11092, 2,908 posts, 195 people, 82 beginner threads) of the public PROHARDVER! topic „Home server / házi szerver építése”, tagged and counted by person (`FINDINGS.md`, method and spot-check in `METHOD.md`; no username or post text in git). Ten ranked recommendations, none acted on; the top three: a „which machine works” page (8 of 33 beginners who name their hardware own only a Synology, a router disk or a Raspberry Pi — the installer cannot run there; the same gap is R-989's first recommendation from a different source, and the shipped-page half is R-991), „you do not choose an operating system — we do” (the biggest beginner question, 20 of 82 threads), and „bring your old office PC” as the cheap entry. Catalog gap (≥ 3 people): a DLNA server (7), an ad/DNS blocker (5), a camera recorder (5). | **WAITING-ON-OPERATOR** — filed 2026-10-12 | the operator's reading | Operator: rule on each of the ten (yes / no / later). If nothing is done: nothing changes; the research ages | operator |
 | **R-1035** | Business & legal | P4 | **Did the „felhő tárhely” page earn anything? Two dated checks.** **Baseline (reported by the operator from Search Console, 2026-10-14, last 3 months):** 1 click, 48 impressions, CTR 2.1 %, average position **73.6**; „felhő tárhely” 19 impressions, „felhőtárhely” 5, „felhő alapú tárhely” 3. CC measured separately that the phrase „felhő tárhely” appeared **0 times** on the home, FAQ and backup pages — Google was showing the home page for those queries by accident. `/sajat-felho-tarhely` + `/en/own-cloud-storage` published **2026-10-14**; full baseline, price sources and the power provenance in `audits/seo-2026-10-14/BASELINE.md`. **What to measure, both times:** impressions and average position for „felhő tárhely” and „saját felhő tárhely”, plus the new page's own impressions, clicks and position. **A null result at +4 weeks means nothing on its own** — a new page on a site at position 73.6 takes weeks to months to move, and „felhő tárhely” alone is held by Microsoft, Google and the big list sites; the page aims at the two longer phrases. That is why there are two checks and not one, and why neither should be read as a verdict on the copy. **There is no Search Console connector in this project, so CC cannot read these numbers** — the operator reads them, CC records and compares. **Blocked until the operator asks for indexing** (Search Console → URL-ellenőrzés → the URL → „Indexelés kérése”): until that happens, or Google finds the page itself, both checks read zero for a reason that is not the page. **This is a slow channel and does not replace talking to people.** | **WATCHING** | operator: request indexing, then supply the Search Console read | 2026-11-11: first read (+4 weeks); 2026-12-09: second (+8 weeks), compared with the first | operator + CC |
 
-## Process & tooling — 34 rows (P3 8, P4 26)
+## Process & tooling — 33 rows (P3 8, P4 25)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -407,5 +408,4 @@ stopping line that lies.
 | R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row |
 <!-- DUE-CHECKS-END -->
 | **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC |
-| **R-1299** | Process & tooling | P4 | **The operator's one-shot clean-up grant (`POST /offsite/window-grant/<customer>`, optional `max_remove`) has no button on any hub page** — the operator could not find it on 2026-10-26 (D55) and had to make a one-purpose access token so CC could call the route. The route, its scope (`offsite:write`) and its event exist (R-833); only the form is missing. | OPEN | — | a small form on the Off-site page (customer, optional cap ≤ 500), with a render test for both branches | CC |
 | **R-1302** | Monitoring & notifications | P4 | **A fresh install's agent logs ERROR „store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" and „capability DEGRADED" for `felhom-pbs` at its FIRST start, although the grant was simply never made yet** — the whole-guest tier is armed (21:00:45) before the PBS storage exists (created 21:00:57 on the hub's descriptor). Seen 2026-10-26 night on the R-1232 BYO box (VM 9375, `audits/night-fixes-2026-10-26/r1232/3-first-run.txt`) and on 2026-10-25's BYO box (`degraded=pve:store-grant:felhom-pbs`). An ERROR that says „investigate the loss" on every new box teaches the reader to skip it. | OPEN | — | on the first grant of a tier whose storage did not exist before, log INFO „granted" instead of the loss ERROR; keep the ERROR for a grant that existed and vanished; a test for both | CC |