WHY .gitignore "was not working": it was working. git never consults
.gitignore for a file it ALREADY TRACKS. The rule `*secret*` matched fine --
proved by dropping an untracked copy in and watching check-ignore name
`.gitignore:3:*secret*`. The file had been tracked since feea0606, which is
ironically the commit that de-gitted the Resend key.
WHAT THE EXPOSED VALUE ACTUALLY WAS. Not an analytics password: the GITEA
ADMIN ACCOUNT PASSWORD (is_admin true; /api/v1/admin/users answered 200), in
a repo gitea.dooplex.hu serves anonymously to the internet. That is push
access to every repo -- including the one whose website/ is git-synced live
and whose scripts/ is published by tag to every new box installer (R-110).
Re-ranked P2 -> P1 on that measurement; my first ranking had only measured
the analytics blast radius.
Every committed value was still live. Nothing had ever been rotated.
ROTATED (values never echoed; written to a 0600 file on DooPlex):
umami-config APP_SECRET + POSTGRES_PASSWORD. The password was
changed INSIDE postgres (ALTER USER) as well as in the
Secret -- the env var is only read at first init, so
patching the Secret alone would have changed nothing.
healthchecks-config SECRET_KEY + SUPERUSER_PASSWORD (nothing consumes them,
there is no healthchecks Deployment).
gitea-creds no longer holds the admin password at all: a SCOPED
token (read:package + read:repository).
gitea admin new random password; gitea-system/gitea-admin updated.
VERIFIED, not assumed:
- new admin password -> 200, OLD PUBLISHED PASSWORD -> 401 (the leak is dead)
- umami: a real beacon returns 200 (so the app authenticates to postgres and
writes) while a bogus site id still returns 400 (so the 200 means something)
- hub: "Registry version check: latest = 0.304.0" AND "Template fetched
(5881 bytes)", no auth failures
- BOTH token scopes are load-bearing, and the second was found by breaking
it: a package-only token made the hub log "Template fetch: unexpected
status 403", because the template fetcher reads a raw file out of the
felhom-controller repo, not the registry.
AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part: the
rollout restart needed to pick up the new umami secret put umami into
CrashLoopBackOff and took stats.felhom.eu down (503) for ~4 minutes. Not the
rotation -- at memory 512Mi that pod runs for months but CANNOT RESTART:
startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137).
Raised to 1Gi IN THE MANIFEST, not just live, per .claude/rules/manifests.md
("never bare kubectl set -- the next sync reverts it and the fix silently
disappears").
THE GATE: KNOWN_BACKLOG is removed from manifest_bearer_gate.py, as its own
comment instructed. Red-proofed with a decoy: exit 1 with it, exit 0 without.
An exemption kept this visible for three months and changed nothing.
WHAT REMAINS (operator, and it is bigger than what was fixed): the same
password is still the admin password in ~12 other namespaces -- nextcloud,
paperless, bookstack (a DATABASE ROOT password), tandoor, calibre,
adventurelog, gokapi, qbittorrent, servarr, homepage. Rotating Gitea does not
touch them. Also owed: a kisfenyo Gitea token sits in plaintext in the local
homelab-manifests remote URL and was printed to a session transcript during
this investigation, so it should be replaced regardless (R-580's shape).
NOT a finding: homelab-manifests is private (404 anonymously) and does not
contain the password; ArgoCD's repo credential is a separate token and was
untouched by the rotation.
umami:
Switch from SHA-pinned v3.0.3 to the tagged v3.1.0 release (the v3
line proper -- same schema lineage, normal Prisma minor-version
migration). This is the documented forward path that the version-
checker hint `postgresql-latest -> 3.1` indicated. The v1.x
postgresql-vX.Y.Z line we briefly tried earlier today is a
DIFFERENT image lineage with incompatible migrations -- avoid.
filebrowser:
Re-pin to v2.63.13 (debian-based default) so Renovate can track
future bumps. The non-root UID in that image can't write to the
existing PVC contents (chowned to root by the previous v2-alpine
image), so set pod-level securityContext runAsUser:0 + runAsGroup:0
to keep using the same volume layout without a chown initContainer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous PR pinned `ghcr.io/umami-software/umami:postgresql-v1.38.0`.
The new pod crashlooped on Prisma:
ERROR: relation "event" does not exist
Migration name: 02_add_event_data
Database error code: 42P01
The 120-day-old working pod's actual image is:
ghcr.io/umami-software/umami@sha256:28f263fe06f79ebffa5a6a6e9b...
It runs an older umami build whose schema doesn't have the `event`
table that the v1 migration `02_add_event_data` operates on. The DB
has migrations 10-14 applied (newer than 02 by name) but 02 isn't in
its applied set -- likely a schema fork between the line our 120d pod
runs and the postgresql-vX.Y.Z line that v1.38.0 advances toward.
Pin to the exact SHA that the working pod uses, so pod restarts +
ArgoCD syncs both keep producing pods on the same known-good image
(cached on the node, no registry pull needed). Renovate also stops
chasing the broken upgrade path.
Proper fix (deferred): plan a v3.x migration. The version-checker
dashboard hint `postgresql-latest → 3.1` suggests umami v3.x dropped
the `postgresql-` prefix and is what we'd want long-term. That needs
a real DB migration plan since the schema lineage is genuinely
different from this image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- umami postgresql-latest -> postgresql-v1.38.0
- filebrowser v2-alpine -> v2.63.13
These two were "latest"-style moving tags that Renovate physically
cannot propose updates for. Pinning to current upstream versions so
future bumps go through the normal Renovate PR flow.
Note: Renovate operates from the homelab-manifests repo, not this one
yet — but felhom-system/* copies exist in homelab-manifests for
discoverability, and Renovate already tracks the pinned forms via a
new customManager for the umami `postgresql-vX.Y.Z` pattern (added in
homelab-manifests admin-system/renovate.yaml). For now, future bumps
will need to be applied to both repos until we consolidate the source
of truth.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
admin