diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index d9db7cc5..5df03f23 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -208,7 +208,7 @@ stopping line that lies. | **R-870** | Security & access | P3 | **Tester 1's two Cloudflare credentials — the zone API token (`infrastructure.cf_api_token`) and the tunnel token (`infrastructure.cf_tunnel_token`) of the hub's `customer_configs` row `tester-1` — were printed into the 2026-10-04 night session's transcript** (not into any file): a read-only query selected `substr(config_json,1,400)`, and both values sit in the first 400 characters. Tester 1 is CC's disposable test customer (`enkicsifelhom.hu`). **Not rotated, by the operator's ruling of 2026-10-05 06:49 (option B).** **Rotation, whenever chosen (3 steps):** in the Cloudflare dashboard create a new API token for the `enkicsifelhom.hu` zone with the same permissions and refresh the Tester 1 tunnel's token (Zero Trust → Networks → Tunnels → the tunnel → refresh token) → hub → Configs → `tester-1` → Edit → the two Cloudflare fields → Save, then confirm on the box that cloudflared reconnected (`docker ps` health `healthy`) → delete the old API token. Rule for sessions (as R-831): never select a whole config row — name the fields, and never `config_json` without `json_extract` of a non-secret field. | **WAITING-ON-OPERATOR — rotation is his call (ruled: not now)** **Not rotated by the operator's rulings (2026-10-04 „keep using the current one"; 2026-10-05 option B) — restated 2026-10-05 18:23; the steps stay here.** | — | rotate when chosen | operator | | **R-908** | Security & access | P3 | **An old Resend API key is still in the history of the `homelab-manifests` repository; it was replaced on 2026-06-29, but whether it is also revoked at Resend is unknown.** FOUND 2026-10-08 by the contact-mailer source search (`audits/mailer-source-2026-10-08/SEARCH.md`, „A side finding"): commit c9648cd (2026-02-05, „added mailer pod") put a key literal in the old `felhom-system/contact-mailer.yaml` comment; the file was deleted in ee93b50 but the history keeps it. Compared by hash only, never printed: it is NOT today's key (`Secret/resend-api`, rotated 2026-06-29, felhom.eu feea0606). If the old key still works at Resend, anyone with read access to that repository can send mail as felhom.eu. | **OPEN — owner: operator** | — | In the Resend console, check that the old key is revoked (revoke it if not); rewriting the repository's history is NOT needed once it is revoked | operator | | **R-525** | Security & access | P4 | **[P3-LOW] FileBrowser has its own login; putting it behind the dashboard session (traefik forwardAuth or Quantum proxy auth) is a new mechanism nobody has measured.** Filed 2026-09-15 by the P1-fixes task (B.5). R-513 closed the default-password hole with a generated password; a household still has two logins. **What it needs:** a spike on a scratch guest — forwardAuth to the controller session, and what FileBrowser Quantum does with a trusted header. | **READY — rank P3-LOW; owner: CC (spike)** **Re-ranked 2026-10-03: P3->P4: comfort feature needing a new unmeasured mechanism; the default password hole is closed.** | — | — | CC | -| **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous** `curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”* **Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config` `APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config` `SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restart**, because startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137). Raised to 1Gi, **in the manifest, not just live** (the `.claude/rules/manifests.md` rule: a bare `kubectl set` is reverted by the next sync and the fix silently disappears). Service restored and verified. New values are in a 0600 file on DooPlex (`~/rotated-secrets-2026-10-09.txt`), never echoed; the operator moves them to the password manager and deletes it. **WHAT REMAINS, and it is bigger than what was fixed:** that one password is still the admin password in about **12 other namespaces** — `nextcloud`, `paperless`, `bookstack` (a **database root** password), `tandoor`, `calibre`, `adventurelog`, `gokapi`, `qbittorrent` (x2), `servarr`, `homepage` (x2). Rotating Gitea does not touch them: each is its own login and each is still the string that was published. **Also owed:** a Gitea token for user `kisfenyo` sits in plaintext in the `origin` URL of the local `homelab-manifests` clone (`.git/config`); CC printed it to a session transcript while investigating, so it should be rotated regardless — this is R-580's shape (store the remote without credentials). **Not a finding:** `homelab-manifests` itself is private (404 anonymously) and does not contain the password; ArgoCD's repo credential is a separate token, so the rotation did not touch it. | **NARROWED 2026-10-09 — the felhom.eu half is DONE and verified (Gitea admin password rotated, old one now 401; umami + healthchecks rotated; file de-gitted; gate carve-out removed). What remains is the ~12 reused logins elsewhere in the homelab and the repo-visibility decision; owner: operator** | — | Operator: **(1)** change the admin password of the ~12 other services that still use the published string (`bookstack` first — it is a database root password); **(2)** rotate the `kisfenyo` Gitea token embedded in the local `homelab-manifests` remote URL, and store the remote without credentials; **(3)** decide whether `gitea.dooplex.hu` should answer anonymously at all — remembering the website git-sync clones it with no credentials and the installer is fetched from a public tag, so making it private breaks both unless they get credentials first, and the 32 `` comments on /adatkezeles become a map of it; **(4)** move `~/rotated-secrets-2026-10-09.txt` into the password manager and delete it. If nothing is done: the published password keeps opening a dozen services, even though Gitea itself is now safe | operator | +| **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous** `curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”* **Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config` `APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config` `SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restart**, because startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137). Raised to 1Gi, **in the manifest, not just live** (the `.claude/rules/manifests.md` rule: a bare `kubectl set` is reverted by the next sync and the fix silently disappears). Service restored and verified. New values are in a 0600 file on DooPlex (`~/rotated-secrets-2026-10-09.txt`), never echoed; the operator moves them to the password manager and deletes it. **WHAT REMAINS, and it is bigger than what was fixed:** that one password is still the admin password in about **12 other namespaces** — `nextcloud`, `paperless`, `bookstack` (a **database root** password), `tandoor`, `calibre`, `adventurelog`, `gokapi`, `qbittorrent` (x2), `servarr`, `homepage` (x2). Rotating Gitea does not touch them: each is its own login and each is still the string that was published. **Also owed:** a Gitea token for user `kisfenyo` sits in plaintext in the `origin` URL of the local `homelab-manifests` clone (`.git/config`); CC printed it to a session transcript while investigating, so it should be rotated regardless — this is R-580's shape (store the remote without credentials). **Not a finding:** `homelab-manifests` itself is private (404 anonymously) and does not contain the password; ArgoCD's repo credential is a separate token, so the rotation did not touch it. | **NARROWED 2026-10-09 — the felhom.eu half is DONE and verified (Gitea admin password rotated, old one now 401; umami + healthchecks rotated; file de-gitted; gate carve-out removed). What remains is the ~12 reused logins elsewhere in the homelab and the repo-visibility decision; owner: operator** | — | **OPERATOR RULED 2026-10-09: (a) he rotates the remaining services himself — CC's scope stopped at the Felhom boundary; (b) the repo STAYS anonymously readable for now**, because making it private breaks the website git-sync and the installer tag fetch, which both clone with no credentials (R-110). The standing consequence of (b): no secret may ever enter this repo again, which `manifest_bearer_gate.py` now enforces with no exemption; and if (b) is reversed, the 32 `` comments on /adatkezeles must be stripped in the same change. **The exact checklist of the 12 remaining secrets** (namespace / secret / key, re-measured after the rotation, with the two traps that bite — a DB password is not changed by editing the Secret, and a pod that has run for months may not restart) is in `documentation/runbooks/secrets.md`. Operator: **(1)** work that checklist, `bookstack-db/root-password` first because it is a database root password and every one of these hosts answers on the public internet; (`bookstack` first — it is a database root password); **(2)** rotate the `kisfenyo` Gitea token embedded in the local `homelab-manifests` remote URL, and store the remote without credentials; **(3)** decide whether `gitea.dooplex.hu` should answer anonymously at all — remembering the website git-sync clones it with no credentials and the installer is fetched from a public tag, so making it private breaks both unless they get credentials first, and the 32 `` comments on /adatkezeles become a map of it; **(4)** move `~/rotated-secrets-2026-10-09.txt` into the password manager and delete it. If nothing is done: the published password keeps opening a dozen services, even though Gitea itself is now safe | operator | | **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator | | **R-904** | Security & access | P4 | **Cloudflare can read every household's app traffic; replacing it with our own relay is a later item.** Facts (reviewer discussion 2026-10-08; `01`): app traffic and the dashboard reach the box through the Cloudflare Tunnel (`01` §5 trust table, rows end-user ↔ apps and customer ↔ controller UI; §7), and the tunnel's public end is Cloudflare's edge, where TLS ends — so Cloudflare can technically read that traffic (the FAQ says so since 2026-10-08, R-900; „TLS ends at the edge" is not written in `01` — add it there). What Cloudflare gives today, free: inbound reach with no router setup, the CGNAT answer (`01` §4, §7); certificates (`01` §7, the free tier covers one level below a zone); the geo-WAF the hub enforces (`01` §5 last row, §7); flood protection (not in `01`). The alternative named: our own EU relay over WireGuard with TLS passthrough by SNI, certificates on the box, the geo-block on the relay. Its costs: one more machine the operator keeps up, and a single point of reach for every box; weaker flood protection; about a week of work after a spike. Operator ruling 2026-10-08 09:07 (`09` §3 decision 184): a later item. | **DEFERRED — after the first customers (operator ruling 2026-10-08 09:07)** | — | A spike after the first customers (the relay's reach, cost and flood behaviour, measured) | operator | | **R-913** | Security & access | P4 | **The Cloudflare token check (R-138, decision 190) reads what a token can SEE, not what it can WRITE.** FOUND 2026-10-08 by the security review of the R-138 build: `GET /zones` lists zones the token can read; a hand-built token with Zone:Read on the customer's zone and DNS:Edit on ALL zones would pass the check and could still change every household's DNS. The token wizard's single „Specific zone" scope does not build such a token, and a customer token cannot read its own policies (that needs „API Tokens Read"). Written as a limit in `01` §7. **Options:** (a) the operator mints every customer token from one recipe and the hub checks nothing more; (b) the hub mints the token itself with the operator's account token (one zone, DNS:Edit) — a new privileged credential on the hub; (c) a negative probe: with the pasted token, try to READ the DNS records of another customer's zone by its id (the hub knows the ids) and refuse on success — catches the read side only. | **OPEN — needs the operator** (a is free today; b is a design) | operator: which of a/b/c | Pick a/b/c; if nothing: the check stands as built and the limit stays written in `01` §7 | operator | diff --git a/documentation/runbooks/secrets.md b/documentation/runbooks/secrets.md index 6e428763..677c22f1 100644 --- a/documentation/runbooks/secrets.md +++ b/documentation/runbooks/secrets.md @@ -222,7 +222,50 @@ sudo kubectl create secret generic gitea-creds -n felhom-system \ # secretKeyRef resend-api/RESEND_API_KEY rather than inlining it (see the Resend section above). ``` -**Still owed, and NOT done here** (they are outside this repo): the same password is still the admin -password in ~12 other namespaces — `nextcloud`, `paperless`, `bookstack` (a **database root** -password), `tandoor`, `calibre`, `adventurelog`, `gokapi`, `qbittorrent`, `servarr`, `homepage`. -Rotating Gitea does not touch those; each is its own login and each is still the published string. +### Still owed — the same password opens 12 more services (operator's, 2026-10-09 ruling) + +Rotating Gitea does **not** touch these: each is its own login and each is still the exact string that +was published. The operator chose to do these himself; CC's scope stopped at the Felhom boundary. + +**Measured 2026-10-09 after the Felhom rotation** (hash comparison against the value still served from +git history — the three CC rotated are confirmed absent from this list): + +``` +[ ] adventurelog-system adventurelog-admin password +[ ] bookstack-system bookstack-db root-password <-- DATABASE root, do first +[ ] calibre-system calibre-auth password +[ ] fileshare-system gokapi-app admin-password +[ ] homepage-system homepage-secrets calibreweb-pass +[ ] homepage-system homepage-secrets qbittorrent-pass +[ ] mediaserver-system qbittorrent-admin password +[ ] nextcloud-system nextcloud nextcloud-password +[ ] paperless-system paperless-admin password +[ ] servarr-system download-client-credentials qbittorrent-password +[ ] servarr-system servarr-credentials password +[ ] tandoor-system tandoor-admin password +``` + +**These are not LAN-only.** `nextcloud`, `paperless`, `bookstack`, `tandoor`, `calibre`, +`adventurelog`, `fileshare`, `plex`, the whole `servarr` set and `homepage` all have +`*.dooplex.hu` ingresses; spot-checked in **public DNS** — `nextcloud/paperless/bookstack/qbittorrent +.dooplex.hu` all resolve to the public address and answer HTTPS with a login page. No login was +attempted; reachability is the point. + +**Two traps when rotating these**, both learned on the Felhom side the same day: + +1. **A database password is not changed by editing the Secret.** `bookstack-db/root-password` is read + when the database is first initialised; afterwards it lives in the database. Change it *in the + engine* and in the Secret, then restart — exactly as `umami-config/POSTGRES_PASSWORD` had to be. +2. **Check the app can still RESTART before you trust it.** Patching a Secret does nothing until the + pod restarts, and a pod that has run for months may not come back: umami ran 124 days at + `512Mi` but was OOMKilled on every restart attempt. Rotate when you can watch it. + +### Repository visibility — DECIDED 2026-10-09: stays public for now + +The operator ruled that `felhom.eu` stays anonymously readable for the moment. Making it private +breaks two live paths that clone it with **no credentials**: the website git-sync in +`manifests/webpage.yaml`, and the installer fetched from the `installer-v…` tag by every new box +(R-110). Both would have to be given credentials first. The standing consequence of that ruling: +**no secret may ever enter this repo again** — which `manifest_bearer_gate.py` now enforces with no +exemption. If the decision is ever reversed, the 32 `` comments served on +`/adatkezeles` must be stripped in the same change, because they are a map of the repo.