Operator ruled 2026-10-09, after seeing the measurement:
(a) he rotates the remaining services himself -- CC's scope stopped at the
Felhom boundary;
(b) felhom.eu STAYS anonymously readable for now, because making it private
breaks the website git-sync and the installer tag fetch, which both
clone with NO credentials (R-110).
The standing consequence of (b): no secret may ever enter this repo again,
which manifest_bearer_gate.py now enforces with no exemption. If (b) is ever
reversed, the 32 <!-- source --> comments served on /adatkezeles must be
stripped in the same change, because they are a map of the repo.
secrets.md now carries the exact checklist -- namespace / secret / key, 12
rows, RE-MEASURED after the Felhom rotation by hashing against the value git
history still serves, which also confirms the three CC rotated are absent
from it.
Two things recorded with it, both learned the hard way the same day:
- a DATABASE password is not changed by editing the Secret. bookstack-db
root-password is read at first init and then lives in the engine, exactly
like umami's POSTGRES_PASSWORD did.
- check the app can still RESTART before trusting the rotation: umami ran
124 days at 512Mi and was OOMKilled on every restart attempt.
And the urgency, measured rather than assumed: these are not LAN-only.
nextcloud / paperless / bookstack / qbittorrent .dooplex.hu all resolve in
PUBLIC DNS to the public address and answer HTTPS with a login page. No login
was attempted; reachability is the point.
WHY .gitignore "was not working": it was working. git never consults
.gitignore for a file it ALREADY TRACKS. The rule `*secret*` matched fine --
proved by dropping an untracked copy in and watching check-ignore name
`.gitignore:3:*secret*`. The file had been tracked since feea0606, which is
ironically the commit that de-gitted the Resend key.
WHAT THE EXPOSED VALUE ACTUALLY WAS. Not an analytics password: the GITEA
ADMIN ACCOUNT PASSWORD (is_admin true; /api/v1/admin/users answered 200), in
a repo gitea.dooplex.hu serves anonymously to the internet. That is push
access to every repo -- including the one whose website/ is git-synced live
and whose scripts/ is published by tag to every new box installer (R-110).
Re-ranked P2 -> P1 on that measurement; my first ranking had only measured
the analytics blast radius.
Every committed value was still live. Nothing had ever been rotated.
ROTATED (values never echoed; written to a 0600 file on DooPlex):
umami-config APP_SECRET + POSTGRES_PASSWORD. The password was
changed INSIDE postgres (ALTER USER) as well as in the
Secret -- the env var is only read at first init, so
patching the Secret alone would have changed nothing.
healthchecks-config SECRET_KEY + SUPERUSER_PASSWORD (nothing consumes them,
there is no healthchecks Deployment).
gitea-creds no longer holds the admin password at all: a SCOPED
token (read:package + read:repository).
gitea admin new random password; gitea-system/gitea-admin updated.
VERIFIED, not assumed:
- new admin password -> 200, OLD PUBLISHED PASSWORD -> 401 (the leak is dead)
- umami: a real beacon returns 200 (so the app authenticates to postgres and
writes) while a bogus site id still returns 400 (so the 200 means something)
- hub: "Registry version check: latest = 0.304.0" AND "Template fetched
(5881 bytes)", no auth failures
- BOTH token scopes are load-bearing, and the second was found by breaking
it: a package-only token made the hub log "Template fetch: unexpected
status 403", because the template fetcher reads a raw file out of the
felhom-controller repo, not the registry.
AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part: the
rollout restart needed to pick up the new umami secret put umami into
CrashLoopBackOff and took stats.felhom.eu down (503) for ~4 minutes. Not the
rotation -- at memory 512Mi that pod runs for months but CANNOT RESTART:
startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137).
Raised to 1Gi IN THE MANIFEST, not just live, per .claude/rules/manifests.md
("never bare kubectl set -- the next sync reverts it and the fix silently
disappears").
THE GATE: KNOWN_BACKLOG is removed from manifest_bearer_gate.py, as its own
comment instructed. Red-proofed with a decoy: exit 1 with it, exit 0 without.
An exemption kept this visible for three months and changed nothing.
WHAT REMAINS (operator, and it is bigger than what was fixed): the same
password is still the admin password in ~12 other namespaces -- nextcloud,
paperless, bookstack (a DATABASE ROOT password), tandoor, calibre,
adventurelog, gokapi, qbittorrent, servarr, homepage. Rotating Gitea does not
touch them. Also owed: a kisfenyo Gitea token sits in plaintext in the local
homelab-manifests remote URL and was printed to a session transcript during
this investigation, so it should be replaced regardless (R-580's shape).
NOT a finding: homelab-manifests is private (404 anonymously) and does not
contain the password; ArgoCD's repo credential is a separate token and was
untouched by the rotation.
Part A evidence (migration spike, sftp-written repo through the pinned rclone key) and the hub
red-proofs under documentation/audits/offsite-lock-build-2026-10-03/. Manifest bump follows after
the image is built and Secret/offsite-secret-key exists.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The global bearer (api.report_api_key) leaves manifests/hub.yaml: the
ConfigMap field is an empty placeholder; the Deployment injects
REPORT_API_KEY from out-of-band Secret/report-api (deliberately NOT
optional - a missing Secret fails Ready rather than booting a hub with an
empty bearer). main.go gains the REPORT_API_KEY env override (the
RESEND_API_KEY twin). scripts/manifest_secret_gate.py blocks bearer-shaped
(64-hex) literals across manifests/ (felhom.secret.yaml's pre-existing
umami APP_SECRET is a visible non-fatal KNOWN-BACKLOG line - secrets.md
backlog, out of scope here).
Ordering (load-bearing): create Secret/report-api with the CURRENT value
BEFORE syncing; this manifest change rides the same sync as the 0.53.0
image (older images ignore the env override). The exposed git-history copy
dies only with the supervised ROTATION - procedure + consumer list in
documentation/runbooks/secrets.md ("Operator/global bearer key"); both
publish runbooks' ROTATION notes now point at that disposition. The
controller repo's example-config copy of the literal is scrubbed
(felhom-controller 51c871a).
Red-proof E: reintroducing a 64-hex literal into manifests/hub.yaml ->
gate exit 1 (recorded); restored clean.
Resend send-scoped key was committed in plaintext in manifests/hub.yaml (ConfigMap)
and manifests/felhom.secret.yaml. Rotated to a new key and removed from git.
- hub: new RESEND_API_KEY env override (cmd/hub/main.go), mirrors REGISTRY_TOKEN;
ConfigMap resend_api_key now an empty placeholder; Deployment injects from
Secret/resend-api. Image 0.17.0.
- contact-mailer: secretKeyRef repointed contact-mailer-config -> resend-api.
- felhom.secret.yaml: contact-mailer-config Secret removed; healthchecks
EMAIL_HOST_PASSWORD blanked (workload not deployed).
- documentation/runbooks/secrets.md: out-of-band secret model + create/rotate steps.
Secret/resend-api is created imperatively out-of-band and is NOT committed.
No secret value appears in this repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin