security: rotate the published secrets and de-git felhom.secret.yaml (R-925, P1)

WHY .gitignore "was not working": it was working. git never consults
.gitignore for a file it ALREADY TRACKS. The rule `*secret*` matched fine --
proved by dropping an untracked copy in and watching check-ignore name
`.gitignore:3:*secret*`. The file had been tracked since feea0606, which is
ironically the commit that de-gitted the Resend key.

WHAT THE EXPOSED VALUE ACTUALLY WAS. Not an analytics password: the GITEA
ADMIN ACCOUNT PASSWORD (is_admin true; /api/v1/admin/users answered 200), in
a repo gitea.dooplex.hu serves anonymously to the internet. That is push
access to every repo -- including the one whose website/ is git-synced live
and whose scripts/ is published by tag to every new box installer (R-110).
Re-ranked P2 -> P1 on that measurement; my first ranking had only measured
the analytics blast radius.

Every committed value was still live. Nothing had ever been rotated.

ROTATED (values never echoed; written to a 0600 file on DooPlex):
  umami-config        APP_SECRET + POSTGRES_PASSWORD. The password was
                      changed INSIDE postgres (ALTER USER) as well as in the
                      Secret -- the env var is only read at first init, so
                      patching the Secret alone would have changed nothing.
  healthchecks-config SECRET_KEY + SUPERUSER_PASSWORD (nothing consumes them,
                      there is no healthchecks Deployment).
  gitea-creds         no longer holds the admin password at all: a SCOPED
                      token (read:package + read:repository).
  gitea admin         new random password; gitea-system/gitea-admin updated.

VERIFIED, not assumed:
  - new admin password -> 200, OLD PUBLISHED PASSWORD -> 401 (the leak is dead)
  - umami: a real beacon returns 200 (so the app authenticates to postgres and
    writes) while a bogus site id still returns 400 (so the 200 means something)
  - hub: "Registry version check: latest = 0.304.0" AND "Template fetched
    (5881 bytes)", no auth failures
  - BOTH token scopes are load-bearing, and the second was found by breaking
    it: a package-only token made the hub log "Template fetch: unexpected
    status 403", because the template fetcher reads a raw file out of the
    felhom-controller repo, not the registry.

AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part: the
rollout restart needed to pick up the new umami secret put umami into
CrashLoopBackOff and took stats.felhom.eu down (503) for ~4 minutes. Not the
rotation -- at memory 512Mi that pod runs for months but CANNOT RESTART:
startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137).
Raised to 1Gi IN THE MANIFEST, not just live, per .claude/rules/manifests.md
("never bare kubectl set -- the next sync reverts it and the fix silently
disappears").

THE GATE: KNOWN_BACKLOG is removed from manifest_bearer_gate.py, as its own
comment instructed. Red-proofed with a decoy: exit 1 with it, exit 0 without.
An exemption kept this visible for three months and changed nothing.

WHAT REMAINS (operator, and it is bigger than what was fixed): the same
password is still the admin password in ~12 other namespaces -- nextcloud,
paperless, bookstack (a DATABASE ROOT password), tandoor, calibre,
adventurelog, gokapi, qbittorrent, servarr, homepage. Rotating Gitea does not
touch them. Also owed: a kisfenyo Gitea token sits in plaintext in the local
homelab-manifests remote URL and was printed to a session transcript during
this investigation, so it should be replaced regardless (R-580's shape).

NOT a finding: homelab-manifests is private (404 anonymously) and does not
contain the password; ArgoCD's repo credential is a separate token and was
untouched by the rotation.

documentation/runbooks/secrets.md

@@ -160,9 +160,69 @@ silently. Rotate the ep0 token with `proxmox-backup-manager user generate-token`
 
 ---
 
-## Other committed secrets (tracked, NOT yet de-gitted — backlog)
+## `felhom.secret.yaml` — ROTATED AND DE-GITTED 2026-10-09 (R-925)
 
-`manifests/felhom.secret.yaml` still commits other plaintext secrets (`healthchecks-config` `SECRET_KEY`
-/ `SUPERUSER_PASSWORD`, `umami-config`, `gitea-creds`). These are **out of scope** for the Resend
-rotation but are the same hygiene problem; de-git them the same way (out-of-band `Secret/...` +
-placeholder) when touched. Tracked here so the gap is visible.
+**What was wrong.** `manifests/felhom.secret.yaml` committed three Secrets in plaintext, and
+`gitea.dooplex.hu` serves this repo to **anyone on the internet with no login** (measured from
+off-network: a real path returns the file, a nonsense path returns 404). Every committed value was
+still live — nothing had ever been rotated. Worst of all, `gitea-creds` was the **Gitea `admin`
+account password** (`is_admin: true`, `/api/v1/admin/users` answered 200), and the *same string* was
+reused as the admin password in **15 cluster Secrets** across the homelab.
+
+**What was done, in this order** (the order matters — de-gitting alone kills nothing, the history
+keeps the value):
+
+1. **`umami-config`** — new random `APP_SECRET` and `POSTGRES_PASSWORD`. The password was changed
+   **inside Postgres** (`ALTER USER umami WITH PASSWORD`) as well as in the Secret; the env var alone
+   would not have changed it, because it is only read when the database is first initialised.
+2. **`healthchecks-config`** — new random `SECRET_KEY` and `SUPERUSER_PASSWORD`. Nothing consumes
+   them (there is no healthchecks Deployment), so this was free.
+3. **`gitea-creds`** — the hub no longer holds the admin password at all. It now holds a **scoped
+   Gitea access token** (`read:package` + `read:repository`). Both scopes are required and were each
+   verified against their real endpoint *before* the swap:
+   `GET /v2/admin/felhom-controller/tags/list` (the registry version checker) and
+   `GET /admin/felhom-controller/raw/branch/main/controller/configs/controller.yaml.example`
+   (the template fetcher — a `read:package`-only token gets **403** here, which is how the missing
+   scope was found).
+4. **The Gitea `admin` password** was changed to a new random value and
+   `gitea-system/gitea-admin` updated. Verified: the new password returns **200**, the old published
+   one returns **401**.
+
+**The values** were written to a 0600 file on DooPlex (`~/rotated-secrets-2026-10-09.txt`) and are to
+be moved into the password manager and the file deleted. They were never echoed to a terminal.
+
+**The file is gone from git** and `.gitignore`'s `*secret*` rule now applies to it (it never did
+before: **`.gitignore` is not consulted for a file git already tracks**, which is why the rule looked
+broken). `scripts/manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out was removed in the same
+commit, so a reintroduction now fails the gate.
+
+### Recreating these Secrets (they are no longer in `manifests/`)
+
+ArgoCD does not manage them any more, exactly like `Secret/resend-api` above. The `felhom` app has
+`automated.enabled: false` and no prune, so removing the file does **not** delete them. To recreate
+one from the password manager, on 192.168.0.180 as `kisfenyo`, writing the value to a 0600 temp file
+first so it is never echoed:
+
+```bash
+# umami-config: APP_SECRET + POSTGRES_PASSWORD.
+# If POSTGRES_PASSWORD changes, ALTER USER in the database too, or umami cannot connect:
+#   kubectl exec -n felhom-system deploy/umami-db -- psql -U umami -d umami \
+#     -c "ALTER USER umami WITH PASSWORD '<new>';"
+sudo kubectl create secret generic umami-config -n felhom-system \
+  --from-file=APP_SECRET=/path/app_secret --from-file=POSTGRES_PASSWORD=/path/pg_pw \
+  --dry-run=client -o yaml | sudo kubectl apply -f -
+
+# gitea-creds: username=admin, password = a SCOPED TOKEN (read:package + read:repository),
+# never the admin account password.
+sudo kubectl create secret generic gitea-creds -n felhom-system \
+  --from-literal=username=admin --from-file=password=/path/token \
+  --dry-run=client -o yaml | sudo kubectl apply -f -
+
+# healthchecks-config: only needed if healthchecks is ever deployed; wire EMAIL_HOST_PASSWORD to
+# secretKeyRef resend-api/RESEND_API_KEY rather than inlining it (see the Resend section above).
+```
+
+**Still owed, and NOT done here** (they are outside this repo): the same password is still the admin
+password in ~12 other namespaces — `nextcloud`, `paperless`, `bookstack` (a **database root**
+password), `tandoor`, `calibre`, `adventurelog`, `gokapi`, `qbittorrent`, `servarr`, `homepage`.
+Rotating Gitea does not touch those; each is its own login and each is still the published string.