security: rotate the published secrets and de-git felhom.secret.yaml (R-925, P1)
gates / gates (push) Successful in 5m23s
gates / gates (push) Successful in 5m23s
WHY .gitignore "was not working": it was working. git never consults
.gitignore for a file it ALREADY TRACKS. The rule `*secret*` matched fine --
proved by dropping an untracked copy in and watching check-ignore name
`.gitignore:3:*secret*`. The file had been tracked since feea0606, which is
ironically the commit that de-gitted the Resend key.
WHAT THE EXPOSED VALUE ACTUALLY WAS. Not an analytics password: the GITEA
ADMIN ACCOUNT PASSWORD (is_admin true; /api/v1/admin/users answered 200), in
a repo gitea.dooplex.hu serves anonymously to the internet. That is push
access to every repo -- including the one whose website/ is git-synced live
and whose scripts/ is published by tag to every new box installer (R-110).
Re-ranked P2 -> P1 on that measurement; my first ranking had only measured
the analytics blast radius.
Every committed value was still live. Nothing had ever been rotated.
ROTATED (values never echoed; written to a 0600 file on DooPlex):
umami-config APP_SECRET + POSTGRES_PASSWORD. The password was
changed INSIDE postgres (ALTER USER) as well as in the
Secret -- the env var is only read at first init, so
patching the Secret alone would have changed nothing.
healthchecks-config SECRET_KEY + SUPERUSER_PASSWORD (nothing consumes them,
there is no healthchecks Deployment).
gitea-creds no longer holds the admin password at all: a SCOPED
token (read:package + read:repository).
gitea admin new random password; gitea-system/gitea-admin updated.
VERIFIED, not assumed:
- new admin password -> 200, OLD PUBLISHED PASSWORD -> 401 (the leak is dead)
- umami: a real beacon returns 200 (so the app authenticates to postgres and
writes) while a bogus site id still returns 400 (so the 200 means something)
- hub: "Registry version check: latest = 0.304.0" AND "Template fetched
(5881 bytes)", no auth failures
- BOTH token scopes are load-bearing, and the second was found by breaking
it: a package-only token made the hub log "Template fetch: unexpected
status 403", because the template fetcher reads a raw file out of the
felhom-controller repo, not the registry.
AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part: the
rollout restart needed to pick up the new umami secret put umami into
CrashLoopBackOff and took stats.felhom.eu down (503) for ~4 minutes. Not the
rotation -- at memory 512Mi that pod runs for months but CANNOT RESTART:
startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137).
Raised to 1Gi IN THE MANIFEST, not just live, per .claude/rules/manifests.md
("never bare kubectl set -- the next sync reverts it and the fix silently
disappears").
THE GATE: KNOWN_BACKLOG is removed from manifest_bearer_gate.py, as its own
comment instructed. Red-proofed with a decoy: exit 1 with it, exit 0 without.
An exemption kept this visible for three months and changed nothing.
WHAT REMAINS (operator, and it is bigger than what was fixed): the same
password is still the admin password in ~12 other namespaces -- nextcloud,
paperless, bookstack (a DATABASE ROOT password), tandoor, calibre,
adventurelog, gokapi, qbittorrent, servarr, homepage. Rotating Gitea does not
touch them. Also owed: a kisfenyo Gitea token sits in plaintext in the local
homelab-manifests remote URL and was printed to a session transcript during
this investigation, so it should be replaced regardless (R-580's shape).
NOT a finding: homelab-manifests is private (404 anonymously) and does not
contain the password; ArgoCD's repo credential is a separate token and was
untouched by the rotation.
This commit is contained in:
@@ -176,7 +176,7 @@
|
||||
| Install-profile gate (shell) | scripts/felhom-host-install.sh `--mode appliance\|byo` (GL-2, v1.10.0) | Mandatory-flag profile (no default), refusals at argv time BEFORE any prompt/step, risky step gated at its CALL SITE (one auditable place — never a branch inside the step), mode persisted to state.json + resume-mismatch refusal, `FELHOM_INSTALL_STATE_DIR` override for harness isolation. Harness: scripts/hostinstall-mode-harness.sh (static refusal matrix + grep-invariants + PVE dry-transcript tier; red-proofs run against a mutated scratch copy). |
|
||||
| Disclosure↔uninstall parity (shell) | scripts/felhom-host-install.sh `_uninstall_statement` + harness GL4-D (v1.11.0) | Every host artifact the byo disclosure names must be removed OR explicitly listed KEPT by `run_uninstall`; the harness greps the parity (token list). New install-time artifact ⇒ add its removal + disclosure line + parity token in the SAME commit. Drive data rule: plain `umount` only, never `-l`/`-f`, never any format op under /mnt/felhom-drives. |
|
||||
| Website deploy (manifest) | manifests/webpage.yaml | git-sync sidecar (sparse-checkout `/website/` + `/scripts/`, `--link=current`) + init container waits for first sync; nginx serves `current/website`; push to main = deployed, no image build. |
|
||||
| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via `secretKeyRef` to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). `report-api` (the operator bearer, v0.53.0) is deliberately NOT `optional:` — a missing Secret fails Ready instead of booting an unauthenticatable hub. `scripts/manifest_bearer_gate.py` (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. ERRATA (2026-07-03): `gitea-creds` is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). |
|
||||
| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via `secretKeyRef` to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). `report-api` (the operator bearer, v0.53.0) is deliberately NOT `optional:` — a missing Secret fails Ready instead of booting an unauthenticatable hub. `scripts/manifest_bearer_gate.py` (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. RESOLVED 2026-10-09 (R-925), was ERRATA 2026-07-03: `gitea-creds` was COMMITTED in the since-deleted `felhom.secret.yaml` manifest while being live-consumed by hub.yaml — and that value was the **Gitea `admin` account password**, in a repo gitea.dooplex.hu serves anonymously to the internet. It is rotated (old password now 401) and the file is removed from git; `gitea-creds` now holds a **scoped token** (`read:package` + `read:repository`), never the admin password. BOTH scopes are load-bearing: the registry version checker needs the first and the template fetcher needs the second, and a package-only token makes the hub log `Template fetch: unexpected status 403`. `KNOWN_BACKLOG` is gone from `manifest_bearer_gate.py`, so a reintroduction now FAILS instead of printing a visible-but-harmless note. Recreate procedure: documentation/runbooks/secrets.md. |
|
||||
| Hub deploy (GitOps) | manifests/hub.yaml `image:` (~L129) | Pinned explicit tag, bumped in git, deliberate ArgoCD sync (auto-sync OFF). Code push alone deploys nothing. |
|
||||
|
||||
## 3. Dangerous lookalikes — do NOT reuse
|
||||
@@ -187,7 +187,7 @@
|
||||
| `(*Handler).handleNotify` + `formatNotificationEmail` + `sendResendEmail` (hub/internal/api/handler.go ~L1289/1624/1589) | Legacy pre-dispatcher notification trio: no cooldowns, no operator channel, no allowedEventTypes gate, duplicate Hungarian formatter. Controller path is FROZEN until slice-10 cutover. | `POST /api/v1/event` → `Dispatcher.ProcessEvent` + `notify.Format*Email` |
|
||||
| Severity `"critical"` POSTed to a PRE-v0.31.0 hub | Fixed in hub v0.31.0 (`handleEvent` now accepts critical). Older hubs coerce `critical` → `"info"`, which never notifies — silent alert loss. Case-variants (`"Critical"`) still coerce on every version. | Against an old hub send `warning`/`error`; otherwise lowercase `critical` is safe |
|
||||
| `compareVersions` for anything security-ish (hub/internal/web/server.go ~L571) | Returns 0 (equal) on unparseable input — a garbage version passes a floor check. `gitea.compareSemver` behaves differently (lexical fallback). | Validate input with `normalizeFloorInput` first; then compareVersions is safe |
|
||||
| Inline `stringData` secrets à la manifests/felhom.secret.yaml | Commits real credentials to git (healthchecks superuser pw, umami APP_SECRET/POSTGRES_PASSWORD, gitea-creds admin password still live there). | Out-of-band `kubectl create secret` + `secretKeyRef` (hub.yaml resend-api pattern; runbook documentation/runbooks/secrets.md) |
|
||||
| Inline `stringData` secrets, as the deleted `felhom.secret.yaml` manifest did | Commits real credentials to git. This was not hypothetical: that file held the healthchecks superuser password, the umami `APP_SECRET`/`POSTGRES_PASSWORD` and — worst — the **Gitea `admin` account password**, in a repo served anonymously to the internet, and every value was still live three months after being flagged. All rotated and the file de-gitted 2026-10-09 (R-925). A `KNOWN_BACKLOG` exemption kept it *visible* in the gate and changed nothing for three months, which is the lesson: an exemption is not a plan. | Out-of-band `kubectl create secret` + `secretKeyRef` (hub.yaml resend-api pattern; runbook documentation/runbooks/secrets.md) |
|
||||
| `kubectl apply` / `kubectl set image` on manifests/ | ArgoCD app `felhom` reverts drift on next sync; live state lies about git. | Edit manifest in git → push → ArgoCD sync (CLAUDE.md steps 3–5) |
|
||||
| `:latest` image tag in manifests | Re-push doesn't change the manifest → no redeploy; Synced/Rollback misreport. | Pinned version tag, bumped per deploy |
|
||||
| The report's `backup` object for snapshots / repo size / integrity (`snapshot_count`, `repo_size_mb`, `integrity_ok`) | **No producer since slice 8C** — the controller's `buildBackupReport` leaves all four zero deliberately and says so. Rendering them gave every customer `Snapshots 0 · Repo Size 0 MB · Integrity Unknown` indefinitely (R-331, measured on demo-hp over a repository holding 67 snapshots). `integrity_ok` is worse than stale: the controller runs no integrity check at all, so it can only ever be "Unknown" or a lie. | The report's `offsite` object (`backup.OffboxReportStatus`) via `reportBackupCard` — and check `stats_known` before trusting a zero |
|
||||
|
||||
@@ -47,7 +47,28 @@
|
||||
- **Still to prove:** the lost off-site copy alarm (Tester 1's next clean-up
|
||||
window, about 12 October), and the failed-restore hold (needs a scratch off-site store).
|
||||
|
||||
## Needs you soon (2026-10-09): passwords are sitting in a repository anyone on the internet can read
|
||||
## Done and still owed (2026-10-09): the published password is dead for Gitea — but it still opens a dozen other things
|
||||
|
||||
Following up the finding below, I measured what that password actually was. It was **your Gitea admin login**,
|
||||
published on the internet. That is push access to every repository — including the one the website is served from
|
||||
and the one new boxes download their installer from. I rotated it, on your instruction.
|
||||
|
||||
- **Dead now:** the old password returns "unauthorized" at Gitea. The visitor counter and the health-check tool
|
||||
got fresh random passwords too. The hub no longer holds your admin password at all — it holds a **limited token**
|
||||
that can only read the things it needs.
|
||||
- **I broke something briefly and fixed it, and you should know:** restarting the visitor counter to pick up its
|
||||
new password took **stats.felhom.eu down for about four minutes**. Not the password — that service had a memory
|
||||
limit it could run under for months but could never *restart* under. It is raised now, in the file as well as on
|
||||
the server, so the next restart works.
|
||||
- **Still open, and this is the bigger half:** the same password is the admin password for about **a dozen other
|
||||
services** — Nextcloud, Paperless, Bookstack (that one is a *database* password), Tandoor, Calibre, qBittorrent
|
||||
and others. Rotating Gitea did nothing for those. Each needs its own new password.
|
||||
- **Also:** a Gitea token of yours sits in plain text inside one local repository's settings, and I printed it to
|
||||
my own session while investigating. Worth replacing.
|
||||
- **Your passwords are in a protected file on DooPlex** (`rotated-secrets-2026-10-09.txt`). Move them into your
|
||||
password manager and delete it.
|
||||
|
||||
## The finding itself (2026-10-09): passwords were sitting in a repository anyone on the internet can read
|
||||
|
||||
I found this while publishing the legal pages, not because I went looking — a security check flagged something
|
||||
small in the new page, and following it led here.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -160,9 +160,69 @@ silently. Rotate the ep0 token with `proxmox-backup-manager user generate-token`
|
||||
|
||||
---
|
||||
|
||||
## Other committed secrets (tracked, NOT yet de-gitted — backlog)
|
||||
## `felhom.secret.yaml` — ROTATED AND DE-GITTED 2026-10-09 (R-925)
|
||||
|
||||
`manifests/felhom.secret.yaml` still commits other plaintext secrets (`healthchecks-config` `SECRET_KEY`
|
||||
/ `SUPERUSER_PASSWORD`, `umami-config`, `gitea-creds`). These are **out of scope** for the Resend
|
||||
rotation but are the same hygiene problem; de-git them the same way (out-of-band `Secret/...` +
|
||||
placeholder) when touched. Tracked here so the gap is visible.
|
||||
**What was wrong.** `manifests/felhom.secret.yaml` committed three Secrets in plaintext, and
|
||||
`gitea.dooplex.hu` serves this repo to **anyone on the internet with no login** (measured from
|
||||
off-network: a real path returns the file, a nonsense path returns 404). Every committed value was
|
||||
still live — nothing had ever been rotated. Worst of all, `gitea-creds` was the **Gitea `admin`
|
||||
account password** (`is_admin: true`, `/api/v1/admin/users` answered 200), and the *same string* was
|
||||
reused as the admin password in **15 cluster Secrets** across the homelab.
|
||||
|
||||
**What was done, in this order** (the order matters — de-gitting alone kills nothing, the history
|
||||
keeps the value):
|
||||
|
||||
1. **`umami-config`** — new random `APP_SECRET` and `POSTGRES_PASSWORD`. The password was changed
|
||||
**inside Postgres** (`ALTER USER umami WITH PASSWORD`) as well as in the Secret; the env var alone
|
||||
would not have changed it, because it is only read when the database is first initialised.
|
||||
2. **`healthchecks-config`** — new random `SECRET_KEY` and `SUPERUSER_PASSWORD`. Nothing consumes
|
||||
them (there is no healthchecks Deployment), so this was free.
|
||||
3. **`gitea-creds`** — the hub no longer holds the admin password at all. It now holds a **scoped
|
||||
Gitea access token** (`read:package` + `read:repository`). Both scopes are required and were each
|
||||
verified against their real endpoint *before* the swap:
|
||||
`GET /v2/admin/felhom-controller/tags/list` (the registry version checker) and
|
||||
`GET /admin/felhom-controller/raw/branch/main/controller/configs/controller.yaml.example`
|
||||
(the template fetcher — a `read:package`-only token gets **403** here, which is how the missing
|
||||
scope was found).
|
||||
4. **The Gitea `admin` password** was changed to a new random value and
|
||||
`gitea-system/gitea-admin` updated. Verified: the new password returns **200**, the old published
|
||||
one returns **401**.
|
||||
|
||||
**The values** were written to a 0600 file on DooPlex (`~/rotated-secrets-2026-10-09.txt`) and are to
|
||||
be moved into the password manager and the file deleted. They were never echoed to a terminal.
|
||||
|
||||
**The file is gone from git** and `.gitignore`'s `*secret*` rule now applies to it (it never did
|
||||
before: **`.gitignore` is not consulted for a file git already tracks**, which is why the rule looked
|
||||
broken). `scripts/manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out was removed in the same
|
||||
commit, so a reintroduction now fails the gate.
|
||||
|
||||
### Recreating these Secrets (they are no longer in `manifests/`)
|
||||
|
||||
ArgoCD does not manage them any more, exactly like `Secret/resend-api` above. The `felhom` app has
|
||||
`automated.enabled: false` and no prune, so removing the file does **not** delete them. To recreate
|
||||
one from the password manager, on 192.168.0.180 as `kisfenyo`, writing the value to a 0600 temp file
|
||||
first so it is never echoed:
|
||||
|
||||
```bash
|
||||
# umami-config: APP_SECRET + POSTGRES_PASSWORD.
|
||||
# If POSTGRES_PASSWORD changes, ALTER USER in the database too, or umami cannot connect:
|
||||
# kubectl exec -n felhom-system deploy/umami-db -- psql -U umami -d umami \
|
||||
# -c "ALTER USER umami WITH PASSWORD '<new>';"
|
||||
sudo kubectl create secret generic umami-config -n felhom-system \
|
||||
--from-file=APP_SECRET=/path/app_secret --from-file=POSTGRES_PASSWORD=/path/pg_pw \
|
||||
--dry-run=client -o yaml | sudo kubectl apply -f -
|
||||
|
||||
# gitea-creds: username=admin, password = a SCOPED TOKEN (read:package + read:repository),
|
||||
# never the admin account password.
|
||||
sudo kubectl create secret generic gitea-creds -n felhom-system \
|
||||
--from-literal=username=admin --from-file=password=/path/token \
|
||||
--dry-run=client -o yaml | sudo kubectl apply -f -
|
||||
|
||||
# healthchecks-config: only needed if healthchecks is ever deployed; wire EMAIL_HOST_PASSWORD to
|
||||
# secretKeyRef resend-api/RESEND_API_KEY rather than inlining it (see the Resend section above).
|
||||
```
|
||||
|
||||
**Still owed, and NOT done here** (they are outside this repo): the same password is still the admin
|
||||
password in ~12 other namespaces — `nextcloud`, `paperless`, `bookstack` (a **database root**
|
||||
password), `tandoor`, `calibre`, `adventurelog`, `gokapi`, `qbittorrent`, `servarr`, `homepage`.
|
||||
Rotating Gitea does not touch those; each is its own login and each is still the published string.
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: healthchecks-config
|
||||
namespace: felhom-system
|
||||
type: Opaque
|
||||
stringData:
|
||||
# === REQUIRED: Generate a random key ===
|
||||
# python3 -c "import secrets; print(secrets.token_urlsafe(50))"
|
||||
SECRET_KEY: "jumZn0XOcO1oDs77siMCgfkg0S2JGLHUiKBAxGleUF0KodBg6SHj-mcLNPxt29Wb6pk"
|
||||
|
||||
# === REQUIRED: Superuser for first login ===
|
||||
SUPERUSER_EMAIL: "admin@felhom.eu"
|
||||
SUPERUSER_PASSWORD: "doodooP4ssWD001!"
|
||||
|
||||
# === REQUIRED: SMTP via Resend.com ===
|
||||
EMAIL_HOST: "smtp.resend.com"
|
||||
EMAIL_PORT: "587"
|
||||
EMAIL_HOST_USER: "resend"
|
||||
# Resend API key — NOT committed. Healthchecks is not currently deployed; when it is, wire its
|
||||
# EMAIL_HOST_PASSWORD to the out-of-band Secret/resend-api (key RESEND_API_KEY) via secretKeyRef
|
||||
# instead of inlining a value here. See documentation/runbooks/secrets.md.
|
||||
EMAIL_HOST_PASSWORD: ""
|
||||
EMAIL_USE_TLS: "True"
|
||||
EMAIL_USE_VERIFICATION: "False"
|
||||
DEFAULT_FROM_EMAIL: "monitoring@felhom.eu"
|
||||
---
|
||||
# NOTE: the Resend API key (formerly Secret/contact-mailer-config RESEND_API_KEY) is no longer
|
||||
# committed. It lives in the out-of-band Secret/resend-api (key RESEND_API_KEY), created imperatively
|
||||
# per documentation/runbooks/secrets.md. Both the hub and contact-mailer now read from resend-api.
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: umami-config
|
||||
namespace: felhom-system
|
||||
type: Opaque
|
||||
stringData:
|
||||
APP_SECRET: "65cee3c4826b2478bcd304d81d3f2193983544d159a7b89dbf2d528479927a86"
|
||||
POSTGRES_PASSWORD: "a764dc950f1f065d8b6f5e402d6420dd"
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: gitea-creds
|
||||
namespace: felhom-system
|
||||
type: Opaque
|
||||
stringData:
|
||||
username: "admin"
|
||||
password: "doodooP4ssWD001!"
|
||||
---
|
||||
@@ -208,12 +208,17 @@ spec:
|
||||
value: "1"
|
||||
- name: TZ
|
||||
value: "Europe/Budapest"
|
||||
# 1Gi, NOT 512Mi (raised 2026-10-09). MEASURED: at 512Mi this pod runs fine for months but
|
||||
# CANNOT RESTART — startup (Prisma migrate + Next.js) peaks above the limit, so the container
|
||||
# reaches "Ready", is OOMKilled (exit 137) and enters CrashLoopBackOff. Found the hard way
|
||||
# during the R-925 secret rotation: a `rollout restart` took stats.felhom.eu down (503) until
|
||||
# the limit was raised. Steady state is well under this; the headroom is for startup only.
|
||||
resources:
|
||||
requests:
|
||||
memory: "128Mi"
|
||||
memory: "256Mi"
|
||||
cpu: "50m"
|
||||
limits:
|
||||
memory: "512Mi"
|
||||
memory: "1Gi"
|
||||
cpu: "500m"
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
|
||||
@@ -21,11 +21,13 @@ ROOT = "manifests"
|
||||
# longer strings still match at 64+, but ordinary short ids never do).
|
||||
BEARER = re.compile(r"(?<![0-9a-fA-F])[0-9a-fA-F]{64}(?![0-9a-fA-F])")
|
||||
|
||||
# KNOWN BACKLOG (non-fatal, stays VISIBLE): felhom.secret.yaml commits pre-existing secrets
|
||||
# (umami APP_SECRET is 64-hex) tracked for de-git in documentation/runbooks/secrets.md — out of
|
||||
# the bearer-de-git scope (2026-07-13 operator ruling batch). Remove this carve-out when that
|
||||
# file is cleaned; new bearer literals must NOT be hidden behind it.
|
||||
KNOWN_BACKLOG = {"felhom.secret.yaml"}
|
||||
# The felhom.secret.yaml carve-out is GONE (2026-10-09, R-925). That file committed the Gitea admin
|
||||
# password, the umami APP_SECRET and more, into a repo gitea.dooplex.hu serves to anyone on the
|
||||
# internet with no login. Every value was rotated and the file was removed from git; `.gitignore`'s
|
||||
# `*secret*` rule applies to it now that it is no longer tracked. The carve-out's own comment said
|
||||
# to remove it when the file was cleaned, so this is that removal — and nothing may be added back
|
||||
# here: an exemption is how the last one stayed visible for three months and changed nothing.
|
||||
KNOWN_BACKLOG = set()
|
||||
|
||||
|
||||
def main():
|
||||
|
||||
Reference in New Issue
Block a user