security: rotate the published secrets and de-git felhom.secret.yaml (R-925, P1)

WHY .gitignore "was not working": it was working. git never consults
.gitignore for a file it ALREADY TRACKS. The rule `*secret*` matched fine --
proved by dropping an untracked copy in and watching check-ignore name
`.gitignore:3:*secret*`. The file had been tracked since feea0606, which is
ironically the commit that de-gitted the Resend key.

WHAT THE EXPOSED VALUE ACTUALLY WAS. Not an analytics password: the GITEA
ADMIN ACCOUNT PASSWORD (is_admin true; /api/v1/admin/users answered 200), in
a repo gitea.dooplex.hu serves anonymously to the internet. That is push
access to every repo -- including the one whose website/ is git-synced live
and whose scripts/ is published by tag to every new box installer (R-110).
Re-ranked P2 -> P1 on that measurement; my first ranking had only measured
the analytics blast radius.

Every committed value was still live. Nothing had ever been rotated.

ROTATED (values never echoed; written to a 0600 file on DooPlex):
  umami-config        APP_SECRET + POSTGRES_PASSWORD. The password was
                      changed INSIDE postgres (ALTER USER) as well as in the
                      Secret -- the env var is only read at first init, so
                      patching the Secret alone would have changed nothing.
  healthchecks-config SECRET_KEY + SUPERUSER_PASSWORD (nothing consumes them,
                      there is no healthchecks Deployment).
  gitea-creds         no longer holds the admin password at all: a SCOPED
                      token (read:package + read:repository).
  gitea admin         new random password; gitea-system/gitea-admin updated.

VERIFIED, not assumed:
  - new admin password -> 200, OLD PUBLISHED PASSWORD -> 401 (the leak is dead)
  - umami: a real beacon returns 200 (so the app authenticates to postgres and
    writes) while a bogus site id still returns 400 (so the 200 means something)
  - hub: "Registry version check: latest = 0.304.0" AND "Template fetched
    (5881 bytes)", no auth failures
  - BOTH token scopes are load-bearing, and the second was found by breaking
    it: a package-only token made the hub log "Template fetch: unexpected
    status 403", because the template fetcher reads a raw file out of the
    felhom-controller repo, not the registry.

AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part: the
rollout restart needed to pick up the new umami secret put umami into
CrashLoopBackOff and took stats.felhom.eu down (503) for ~4 minutes. Not the
rotation -- at memory 512Mi that pod runs for months but CANNOT RESTART:
startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137).
Raised to 1Gi IN THE MANIFEST, not just live, per .claude/rules/manifests.md
("never bare kubectl set -- the next sync reverts it and the fix silently
disappears").

THE GATE: KNOWN_BACKLOG is removed from manifest_bearer_gate.py, as its own
comment instructed. Red-proofed with a decoy: exit 1 with it, exit 0 without.
An exemption kept this visible for three months and changed nothing.

WHAT REMAINS (operator, and it is bigger than what was fixed): the same
password is still the admin password in ~12 other namespaces -- nextcloud,
paperless, bookstack (a DATABASE ROOT password), tandoor, calibre,
adventurelog, gokapi, qbittorrent, servarr, homepage. Rotating Gitea does not
touch them. Also owed: a kisfenyo Gitea token sits in plaintext in the local
homelab-manifests remote URL and was printed to a session transcript during
this investigation, so it should be replaced regardless (R-580's shape).

NOT a finding: homelab-manifests is private (404 anonymously) and does not
contain the password; ArgoCD's repo credential is a separate token and was
untouched by the rotation.

REUSE.md

@@ -176,7 +176,7 @@
 | Install-profile gate (shell) | scripts/felhom-host-install.sh `--mode appliance\|byo` (GL-2, v1.10.0) | Mandatory-flag profile (no default), refusals at argv time BEFORE any prompt/step, risky step gated at its CALL SITE (one auditable place — never a branch inside the step), mode persisted to state.json + resume-mismatch refusal, `FELHOM_INSTALL_STATE_DIR` override for harness isolation. Harness: scripts/hostinstall-mode-harness.sh (static refusal matrix + grep-invariants + PVE dry-transcript tier; red-proofs run against a mutated scratch copy). |
 | Disclosure↔uninstall parity (shell) | scripts/felhom-host-install.sh `_uninstall_statement` + harness GL4-D (v1.11.0) | Every host artifact the byo disclosure names must be removed OR explicitly listed KEPT by `run_uninstall`; the harness greps the parity (token list). New install-time artifact ⇒ add its removal + disclosure line + parity token in the SAME commit. Drive data rule: plain `umount` only, never `-l`/`-f`, never any format op under /mnt/felhom-drives. |
 | Website deploy (manifest) | manifests/webpage.yaml | git-sync sidecar (sparse-checkout `/website/` + `/scripts/`, `--link=current`) + init container waits for first sync; nginx serves `current/website`; push to main = deployed, no image build. |
-| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via `secretKeyRef` to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). `report-api` (the operator bearer, v0.53.0) is deliberately NOT `optional:` — a missing Secret fails Ready instead of booting an unauthenticatable hub. `scripts/manifest_bearer_gate.py` (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. ERRATA (2026-07-03): `gitea-creds` is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). |
+| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via `secretKeyRef` to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). `report-api` (the operator bearer, v0.53.0) is deliberately NOT `optional:` — a missing Secret fails Ready instead of booting an unauthenticatable hub. `scripts/manifest_bearer_gate.py` (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. RESOLVED 2026-10-09 (R-925), was ERRATA 2026-07-03: `gitea-creds` was COMMITTED in the since-deleted `felhom.secret.yaml` manifest while being live-consumed by hub.yaml — and that value was the **Gitea `admin` account password**, in a repo gitea.dooplex.hu serves anonymously to the internet. It is rotated (old password now 401) and the file is removed from git; `gitea-creds` now holds a **scoped token** (`read:package` + `read:repository`), never the admin password. BOTH scopes are load-bearing: the registry version checker needs the first and the template fetcher needs the second, and a package-only token makes the hub log `Template fetch: unexpected status 403`. `KNOWN_BACKLOG` is gone from `manifest_bearer_gate.py`, so a reintroduction now FAILS instead of printing a visible-but-harmless note. Recreate procedure: documentation/runbooks/secrets.md. |
 | Hub deploy (GitOps) | manifests/hub.yaml `image:` (~L129) | Pinned explicit tag, bumped in git, deliberate ArgoCD sync (auto-sync OFF). Code push alone deploys nothing. |
 
 ## 3. Dangerous lookalikes — do NOT reuse
@@ -187,7 +187,7 @@
 | `(*Handler).handleNotify` + `formatNotificationEmail` + `sendResendEmail` (hub/internal/api/handler.go ~L1289/1624/1589) | Legacy pre-dispatcher notification trio: no cooldowns, no operator channel, no allowedEventTypes gate, duplicate Hungarian formatter. Controller path is FROZEN until slice-10 cutover. | `POST /api/v1/event` → `Dispatcher.ProcessEvent` + `notify.Format*Email` |
 | Severity `"critical"` POSTed to a PRE-v0.31.0 hub | Fixed in hub v0.31.0 (`handleEvent` now accepts critical). Older hubs coerce `critical` → `"info"`, which never notifies — silent alert loss. Case-variants (`"Critical"`) still coerce on every version. | Against an old hub send `warning`/`error`; otherwise lowercase `critical` is safe |
 | `compareVersions` for anything security-ish (hub/internal/web/server.go ~L571) | Returns 0 (equal) on unparseable input — a garbage version passes a floor check. `gitea.compareSemver` behaves differently (lexical fallback). | Validate input with `normalizeFloorInput` first; then compareVersions is safe |
-| Inline `stringData` secrets à la manifests/felhom.secret.yaml | Commits real credentials to git (healthchecks superuser pw, umami APP_SECRET/POSTGRES_PASSWORD, gitea-creds admin password still live there). | Out-of-band `kubectl create secret` + `secretKeyRef` (hub.yaml resend-api pattern; runbook documentation/runbooks/secrets.md) |
+| Inline `stringData` secrets, as the deleted `felhom.secret.yaml` manifest did | Commits real credentials to git. This was not hypothetical: that file held the healthchecks superuser password, the umami `APP_SECRET`/`POSTGRES_PASSWORD` and — worst — the **Gitea `admin` account password**, in a repo served anonymously to the internet, and every value was still live three months after being flagged. All rotated and the file de-gitted 2026-10-09 (R-925). A `KNOWN_BACKLOG` exemption kept it *visible* in the gate and changed nothing for three months, which is the lesson: an exemption is not a plan. | Out-of-band `kubectl create secret` + `secretKeyRef` (hub.yaml resend-api pattern; runbook documentation/runbooks/secrets.md) |
 | `kubectl apply` / `kubectl set image` on manifests/ | ArgoCD app `felhom` reverts drift on next sync; live state lies about git. | Edit manifest in git → push → ArgoCD sync (CLAUDE.md steps 3–5) |
 | `:latest` image tag in manifests | Re-push doesn't change the manifest → no redeploy; Synced/Rollback misreport. | Pinned version tag, bumped per deploy |
 | The report's `backup` object for snapshots / repo size / integrity (`snapshot_count`, `repo_size_mb`, `integrity_ok`) | **No producer since slice 8C** — the controller's `buildBackupReport` leaves all four zero deliberately and says so. Rendering them gave every customer `Snapshots 0 · Repo Size 0 MB · Integrity Unknown` indefinitely (R-331, measured on demo-hp over a repository holding 67 snapshots). `integrity_ok` is worse than stale: the controller runs no integrity check at all, so it can only ever be "Unknown" or a lie. | The report's `offsite` object (`backup.OffboxReportStatus`) via `reportBackupCard` — and check `stats_known` before trusting a zero |

STATUS.md

@@ -47,7 +47,28 @@
 - **Still to prove:** the lost off-site copy alarm (Tester 1's next clean-up
   window, about 12 October), and the failed-restore hold (needs a scratch off-site store).
 
-## Needs you soon (2026-10-09): passwords are sitting in a repository anyone on the internet can read
+## Done and still owed (2026-10-09): the published password is dead for Gitea — but it still opens a dozen other things
+
+Following up the finding below, I measured what that password actually was. It was **your Gitea admin login**,
+published on the internet. That is push access to every repository — including the one the website is served from
+and the one new boxes download their installer from. I rotated it, on your instruction.
+
+- **Dead now:** the old password returns "unauthorized" at Gitea. The visitor counter and the health-check tool
+  got fresh random passwords too. The hub no longer holds your admin password at all — it holds a **limited token**
+  that can only read the things it needs.
+- **I broke something briefly and fixed it, and you should know:** restarting the visitor counter to pick up its
+  new password took **stats.felhom.eu down for about four minutes**. Not the password — that service had a memory
+  limit it could run under for months but could never *restart* under. It is raised now, in the file as well as on
+  the server, so the next restart works.
+- **Still open, and this is the bigger half:** the same password is the admin password for about **a dozen other
+  services** — Nextcloud, Paperless, Bookstack (that one is a *database* password), Tandoor, Calibre, qBittorrent
+  and others. Rotating Gitea did nothing for those. Each needs its own new password.
+- **Also:** a Gitea token of yours sits in plain text inside one local repository's settings, and I printed it to
+  my own session while investigating. Worth replacing.
+- **Your passwords are in a protected file on DooPlex** (`rotated-secrets-2026-10-09.txt`). Move them into your
+  password manager and delete it.
+
+## The finding itself (2026-10-09): passwords were sitting in a repository anyone on the internet can read
 
 I found this while publishing the legal pages, not because I went looking — a security check flagged something
 small in the new page, and following it led here.

documentation/runbooks/secrets.md

@@ -160,9 +160,69 @@ silently. Rotate the ep0 token with `proxmox-backup-manager user generate-token`
 
 ---
 
-## Other committed secrets (tracked, NOT yet de-gitted — backlog)
+## `felhom.secret.yaml` — ROTATED AND DE-GITTED 2026-10-09 (R-925)
 
-`manifests/felhom.secret.yaml` still commits other plaintext secrets (`healthchecks-config` `SECRET_KEY`
+**What was wrong.** `manifests/felhom.secret.yaml` committed three Secrets in plaintext, and
-/ `SUPERUSER_PASSWORD`, `umami-config`, `gitea-creds`). These are **out of scope** for the Resend
+`gitea.dooplex.hu` serves this repo to **anyone on the internet with no login** (measured from
-rotation but are the same hygiene problem; de-git them the same way (out-of-band `Secret/...` +
+off-network: a real path returns the file, a nonsense path returns 404). Every committed value was
-placeholder) when touched. Tracked here so the gap is visible.
+still live — nothing had ever been rotated. Worst of all, `gitea-creds` was the **Gitea `admin`
+account password** (`is_admin: true`, `/api/v1/admin/users` answered 200), and the *same string* was
+reused as the admin password in **15 cluster Secrets** across the homelab.
+
+**What was done, in this order** (the order matters — de-gitting alone kills nothing, the history
+keeps the value):
+
+1. **`umami-config`** — new random `APP_SECRET` and `POSTGRES_PASSWORD`. The password was changed
+   **inside Postgres** (`ALTER USER umami WITH PASSWORD`) as well as in the Secret; the env var alone
+   would not have changed it, because it is only read when the database is first initialised.
+2. **`healthchecks-config`** — new random `SECRET_KEY` and `SUPERUSER_PASSWORD`. Nothing consumes
+   them (there is no healthchecks Deployment), so this was free.
+3. **`gitea-creds`** — the hub no longer holds the admin password at all. It now holds a **scoped
+   Gitea access token** (`read:package` + `read:repository`). Both scopes are required and were each
+   verified against their real endpoint *before* the swap:
+   `GET /v2/admin/felhom-controller/tags/list` (the registry version checker) and
+   `GET /admin/felhom-controller/raw/branch/main/controller/configs/controller.yaml.example`
+   (the template fetcher — a `read:package`-only token gets **403** here, which is how the missing
+   scope was found).
+4. **The Gitea `admin` password** was changed to a new random value and
+   `gitea-system/gitea-admin` updated. Verified: the new password returns **200**, the old published
+   one returns **401**.
+
+**The values** were written to a 0600 file on DooPlex (`~/rotated-secrets-2026-10-09.txt`) and are to
+be moved into the password manager and the file deleted. They were never echoed to a terminal.
+
+**The file is gone from git** and `.gitignore`'s `*secret*` rule now applies to it (it never did
+before: **`.gitignore` is not consulted for a file git already tracks**, which is why the rule looked
+broken). `scripts/manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out was removed in the same
+commit, so a reintroduction now fails the gate.
+
+### Recreating these Secrets (they are no longer in `manifests/`)
+
+ArgoCD does not manage them any more, exactly like `Secret/resend-api` above. The `felhom` app has
+`automated.enabled: false` and no prune, so removing the file does **not** delete them. To recreate
+one from the password manager, on 192.168.0.180 as `kisfenyo`, writing the value to a 0600 temp file
+first so it is never echoed:
+
+```bash
+# umami-config: APP_SECRET + POSTGRES_PASSWORD.
+# If POSTGRES_PASSWORD changes, ALTER USER in the database too, or umami cannot connect:
+#   kubectl exec -n felhom-system deploy/umami-db -- psql -U umami -d umami \
+#     -c "ALTER USER umami WITH PASSWORD '<new>';"
+sudo kubectl create secret generic umami-config -n felhom-system \
+  --from-file=APP_SECRET=/path/app_secret --from-file=POSTGRES_PASSWORD=/path/pg_pw \
+  --dry-run=client -o yaml | sudo kubectl apply -f -
+
+# gitea-creds: username=admin, password = a SCOPED TOKEN (read:package + read:repository),
+# never the admin account password.
+sudo kubectl create secret generic gitea-creds -n felhom-system \
+  --from-literal=username=admin --from-file=password=/path/token \
+  --dry-run=client -o yaml | sudo kubectl apply -f -
+
+# healthchecks-config: only needed if healthchecks is ever deployed; wire EMAIL_HOST_PASSWORD to
+# secretKeyRef resend-api/RESEND_API_KEY rather than inlining it (see the Resend section above).
+```
+
+**Still owed, and NOT done here** (they are outside this repo): the same password is still the admin
+password in ~12 other namespaces — `nextcloud`, `paperless`, `bookstack` (a **database root**
+password), `tandoor`, `calibre`, `adventurelog`, `gokapi`, `qbittorrent`, `servarr`, `homepage`.
+Rotating Gitea does not touch those; each is its own login and each is still the published string.

manifests/felhom.secret.yaml

@@ -1,51 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-  name: healthchecks-config
-  namespace: felhom-system
-type: Opaque
-stringData:
-  # === REQUIRED: Generate a random key ===
-  # python3 -c "import secrets; print(secrets.token_urlsafe(50))"
-  SECRET_KEY: "jumZn0XOcO1oDs77siMCgfkg0S2JGLHUiKBAxGleUF0KodBg6SHj-mcLNPxt29Wb6pk"
-
-  # === REQUIRED: Superuser for first login ===
-  SUPERUSER_EMAIL: "admin@felhom.eu"
-  SUPERUSER_PASSWORD: "doodooP4ssWD001!"
-
-  # === REQUIRED: SMTP via Resend.com ===
-  EMAIL_HOST: "smtp.resend.com"
-  EMAIL_PORT: "587"
-  EMAIL_HOST_USER: "resend"
-  # Resend API key — NOT committed. Healthchecks is not currently deployed; when it is, wire its
-  # EMAIL_HOST_PASSWORD to the out-of-band Secret/resend-api (key RESEND_API_KEY) via secretKeyRef
-  # instead of inlining a value here. See documentation/runbooks/secrets.md.
-  EMAIL_HOST_PASSWORD: ""
-  EMAIL_USE_TLS: "True"
-  EMAIL_USE_VERIFICATION: "False"
-  DEFAULT_FROM_EMAIL: "monitoring@felhom.eu"
----
-# NOTE: the Resend API key (formerly Secret/contact-mailer-config RESEND_API_KEY) is no longer
-# committed. It lives in the out-of-band Secret/resend-api (key RESEND_API_KEY), created imperatively
-# per documentation/runbooks/secrets.md. Both the hub and contact-mailer now read from resend-api.
----
-apiVersion: v1
-kind: Secret
-metadata:
-  name: umami-config
-  namespace: felhom-system
-type: Opaque
-stringData:
-  APP_SECRET: "65cee3c4826b2478bcd304d81d3f2193983544d159a7b89dbf2d528479927a86"
-  POSTGRES_PASSWORD: "a764dc950f1f065d8b6f5e402d6420dd"
----
-apiVersion: v1
-kind: Secret
-metadata:
-  name: gitea-creds
-  namespace: felhom-system
-type: Opaque
-stringData:
-  username: "admin"
-  password: "doodooP4ssWD001!"
----

manifests/umami.yaml

@@ -208,12 +208,17 @@ spec:
               value: "1"
             - name: TZ
               value: "Europe/Budapest"
+          # 1Gi, NOT 512Mi (raised 2026-10-09). MEASURED: at 512Mi this pod runs fine for months but
+          # CANNOT RESTART — startup (Prisma migrate + Next.js) peaks above the limit, so the container
+          # reaches "Ready", is OOMKilled (exit 137) and enters CrashLoopBackOff. Found the hard way
+          # during the R-925 secret rotation: a `rollout restart` took stats.felhom.eu down (503) until
+          # the limit was raised. Steady state is well under this; the headroom is for startup only.
           resources:
             requests:
-              memory: "128Mi"
+              memory: "256Mi"
               cpu: "50m"
             limits:
-              memory: "512Mi"
+              memory: "1Gi"
               cpu: "500m"
           livenessProbe:
             httpGet:

scripts/manifest_bearer_gate.py

@@ -21,11 +21,13 @@ ROOT = "manifests"
 # longer strings still match at 64+, but ordinary short ids never do).
 BEARER = re.compile(r"(?<![0-9a-fA-F])[0-9a-fA-F]{64}(?![0-9a-fA-F])")
 
-# KNOWN BACKLOG (non-fatal, stays VISIBLE): felhom.secret.yaml commits pre-existing secrets
+# The felhom.secret.yaml carve-out is GONE (2026-10-09, R-925). That file committed the Gitea admin
-# (umami APP_SECRET is 64-hex) tracked for de-git in documentation/runbooks/secrets.md — out of
+# password, the umami APP_SECRET and more, into a repo gitea.dooplex.hu serves to anyone on the
-# the bearer-de-git scope (2026-07-13 operator ruling batch). Remove this carve-out when that
+# internet with no login. Every value was rotated and the file was removed from git; `.gitignore`'s
-# file is cleaned; new bearer literals must NOT be hidden behind it.
+# `*secret*` rule applies to it now that it is no longer tracked. The carve-out's own comment said
-KNOWN_BACKLOG = {"felhom.secret.yaml"}
+# to remove it when the file was cleaned, so this is that removal — and nothing may be added back
+# here: an exemption is how the last one stayed visible for three months and changed nothing.
+KNOWN_BACKLOG = set()
 
 
 def main():