A heal marker is an event, not a baseline: construction seeds pre-existing markers
(startup false-alarm guard) but a newly-observed marker now raises the warning, so
the first auto-heal surfaces (matches the live drill). Added tests for both halves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Peersync script: validate-first (jq contract check before ANY state change),
head-file + generated-peers conf model, syncconf-from-tmp then atomic mv (live
conf never diverges in the failure direction), zero-peer payload valid (wipe).
hub.yaml: 0.32.0 image + WG_ENDPOINT_SSH_* env + optional Secret mount so the
pod starts before the runbook's step-6 Secret exists.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Part A (hub): move the global-floor + Day-0-artifacts cards from the Customers
page to the Configuration tab; routes → /configuration/{global-floor,artifacts};
redirects + flashes to /configuration. Customers page back to list + Add.
Part B: online setup command on the customer page (download-then-run, passphrase
at prompt, not templated); serve /scripts/ from the website (sparse-checkout +
nginx location) so felhom.eu/scripts/felhom-host-install.sh resolves; script
passphrase prompt reads < /dev/tty (works for pipe-to-bash too).
Part C (script): --vmid auto-detect — default 9201 in use + no --force → pick the
next free id from pct+qm and confirm; explicit --vmid stays die-unless-force.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dispatcher's operator path was gated off (processOperator returns
unless operatorOn && operatorEmail!=""), so the self-health pipeline
stopped one hop short of the inbox. Set the operator recipient
(admin@felhom.eu — the operator's own address, not a secret) and enable.
Config-only; no image bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
Resend send-scoped key was committed in plaintext in manifests/hub.yaml (ConfigMap)
and manifests/felhom.secret.yaml. Rotated to a new key and removed from git.
- hub: new RESEND_API_KEY env override (cmd/hub/main.go), mirrors REGISTRY_TOKEN;
ConfigMap resend_api_key now an empty placeholder; Deployment injects from
Secret/resend-api. Image 0.17.0.
- contact-mailer: secretKeyRef repointed contact-mailer-config -> resend-api.
- felhom.secret.yaml: contact-mailer-config Secret removed; healthchecks
EMAIL_HOST_PASSWORD blanked (workload not deployed).
- documentation/runbooks/secrets.md: out-of-band secret model + create/rotate steps.
Secret/resend-api is created imperatively out-of-band and is NOT committed.
No secret value appears in this repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- cmd/hub/main.go: seed the Day-0 artifact manifest from ARTIFACT_AGENT_VERSION/
ARTIFACT_AGENT_SHA256/ARTIFACT_GOLDEN_VERSION/ARTIFACT_GOLDEN_SHA256 on startup
(only empty fields, so UI edits stick) — same escape hatch the floor uses.
- manifests/hub.yaml: image 0.15.0 -> 0.16.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DR recipe slice (hub half), grounded in SPIKE-dr-recipe-2026-06-16. The hub
receives two additive dr_recipe halves on the existing report paths (agent
storage/guest/PBS on host-report; controller customer/apps on the controller
report), stores them PLAINTEXT in a DEDICATED dr_recipe table keyed by customer
(each half preserves the other), and AssembleDRRecipe stitches them into one
operator-readable recipe (ignore-unknown + version-skew tolerant).
View: a DR-recipe panel on the customer page + GET /customers/{id}/dr-recipe.json
download (operator-auth, no secrets to redact). Plaintext-at-rest is correct —
the recipe is the clean inverse of the retired infra-backup.
Tests: store round-trip (each half preserves the other), assemble-matches-golden,
ignore-unknown + version skew, partial halves, no-secrets sweep. Manifest tag
bumped to v0.13.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase-1 of SPIKE-infra-backup-2026-06-15. The infra-backup mechanism was dead
since slice 8C yet stored plaintext customer secrets at rest (app-secret key,
restic password, Cloudflare tokens) — a zero-knowledge violation — and its
absence made the daily expected_backup_missed email fire for healthy customers.
- Repoint monitor.CheckBackupDeadlines backup half to the agent host-report's
PBS snapshots (+vzdump): alarm only on no-backup / >26h stale / verify failed.
Keep the db_dump half. No host-report → no backup alarm (liveness owns that).
New store.GetLatestHostReportJSON. Tests incl. a companion that fails pre-fix.
- Remove the infra-backup endpoints, store methods/types, and operator panel;
/recovery now returns config_yaml only.
- migrate(): DROP infra_backup_versions/infra_backups + VACUUM (+wal_checkpoint)
to physically reclaim the plaintext pages, gated on table existence.
Flagged out-of-scope: exposed creds need operator rotation; legacy reports table
holds historical plaintext restic_password rows (separate leak, not purged here).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live hub was v0.6.3 (pre host-report endpoint); v0.7.0-v0.7.2 were changelogged but
the manifest was never bumped. This deploys the host-domain ingest (slice 3) +
storage_targets (slice 5 Phase A). Additive/idempotent migrate(); controller path untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous PR pinned filebrowser to v2.63.13 + runAsUser:0 which
solved the PVC permission issue, but the pod was still 0/1 Ready
because v2.63.x changed the default config-file lookup path:
Old (v2-alpine): /.filebrowser.json (matched our existing mount)
New (v2.63.13) : /config/settings.json (NOT mounted in this pod)
So the new image ran with its built-in defaults (port 80, in-memory
db), and the readiness probe on 8080/health timed out.
Fix: pass `args: ["-c", "/.filebrowser.json"]` so filebrowser uses the
ConfigMap we already mount there. No volumeMount changes needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
umami:
Switch from SHA-pinned v3.0.3 to the tagged v3.1.0 release (the v3
line proper -- same schema lineage, normal Prisma minor-version
migration). This is the documented forward path that the version-
checker hint `postgresql-latest -> 3.1` indicated. The v1.x
postgresql-vX.Y.Z line we briefly tried earlier today is a
DIFFERENT image lineage with incompatible migrations -- avoid.
filebrowser:
Re-pin to v2.63.13 (debian-based default) so Renovate can track
future bumps. The non-root UID in that image can't write to the
existing PVC contents (chowned to root by the previous v2-alpine
image), so set pod-level securityContext runAsUser:0 + runAsGroup:0
to keep using the same volume layout without a chown initContainer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous PR pinned `ghcr.io/umami-software/umami:postgresql-v1.38.0`.
The new pod crashlooped on Prisma:
ERROR: relation "event" does not exist
Migration name: 02_add_event_data
Database error code: 42P01
The 120-day-old working pod's actual image is:
ghcr.io/umami-software/umami@sha256:28f263fe06f79ebffa5a6a6e9b...
It runs an older umami build whose schema doesn't have the `event`
table that the v1 migration `02_add_event_data` operates on. The DB
has migrations 10-14 applied (newer than 02 by name) but 02 isn't in
its applied set -- likely a schema fork between the line our 120d pod
runs and the postgresql-vX.Y.Z line that v1.38.0 advances toward.
Pin to the exact SHA that the working pod uses, so pod restarts +
ArgoCD syncs both keep producing pods on the same known-good image
(cached on the node, no registry pull needed). Renovate also stops
chasing the broken upgrade path.
Proper fix (deferred): plan a v3.x migration. The version-checker
dashboard hint `postgresql-latest → 3.1` suggests umami v3.x dropped
the `postgresql-` prefix and is what we'd want long-term. That needs
a real DB migration plan since the schema lineage is genuinely
different from this image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous PR pinned `filebrowser/filebrowser:v2-alpine` to v2.63.13
but it crashlooped on:
Error: open /database/filebrowser.db: permission denied
The v2.63.13 image (debian-based default) runs as a non-root UID and
can't write to files on the PVC that were created by the v2-alpine
image (which ran as root). No `v2.63.13-alpine` tag exists upstream
(filebrowser stopped publishing per-version alpine variants), so we
can't trivially preserve the same runtime.
Quick recovery: revert to v2-alpine so filebrowser is usable again.
Proper fix (deferred): either an initContainer that `chown -R 1000:1000
/database /srv` or a `securityContext.fsGroup: 1000` on the pod spec
to let the non-root UID write to the existing PVC. Both require some
care since the chown is destructive if the UID is wrong.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- umami postgresql-latest -> postgresql-v1.38.0
- filebrowser v2-alpine -> v2.63.13
These two were "latest"-style moving tags that Renovate physically
cannot propose updates for. Pinning to current upstream versions so
future bumps go through the normal Renovate PR flow.
Note: Renovate operates from the homelab-manifests repo, not this one
yet — but felhom-system/* copies exist in homelab-manifests for
discoverability, and Renovate already tracks the pinned forms via a
new customManager for the umami `postgresql-vX.Y.Z` pattern (added in
homelab-manifests admin-system/renovate.yaml). For now, future bumps
will need to be applied to both repos until we consolidate the source
of truth.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
admin