Files
felhom.eu/REUSE.md
T
admin 6218e7919d hub v0.56.0: PBS-DR self-heal reconciler (re-stage a consumable secret)
Implements SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44). A box re-installed/rolled
back onto its stable host_id loses its agent-side converged marker; the hub
keeps the enabled descriptor + a CONSUMED one-time secret, the WG peer persists
(changed==false, cascade can't re-fire), so the agent sits in waiting_secret
forever. The missing piece is a consumable secret, not the descriptor.

New internal/pbsdrheal reconciler (5m, wgsync shape): for enabled+provisioned
hosts whose latest report pbs_dr.state is a stuck state past a >=2-distinct-report
debounce, re-stage the stored secret (store.RestageHostPBSSecret: clear
consumed_at, no ep0 call, NO generation bump); escalate to Re-issue (web
ReissuePBSDR) only when no secret is stored or the agent reports consumed_failed.
Converged/disabled/verify_failed/DR-OFF = no-op. PBSDRHEAL_ONLY_HOST scopes a
supervised rollout. Scenarios A-F + all six red-proofs verified. No agent change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
2026-07-15 18:27:39 +02:00

35 KiB
Raw Blame History

REUSE.md — felhom.eu (hub + website + scripts)

Before writing new code, check here. Canonical helpers, patterns to copy, traps to avoid. Maintenance: update in the SAME commit that adds/changes/deprecates a shared helper. Entries cite file + symbol. Line numbers are landmarks only — reconfirm before editing.

1. Canonical helpers (MUST reuse — do not reinvent)

Report ingest & API auth (hub/internal/api/)

Symbol File Short signature Use for Gotchas
(*Handler).checkAuthCustomer hub/internal/api/handler.go (~L94) (r) (customerID string, isGlobal, ok bool) Bearer auth for controller-facing endpoints (global key OR per-customer key) Global key → ("", true, true): caller must then trust body customer_id. Constant-time compare on global key.
(*Handler).checkAuthHost hub/internal/api/handler.go (~L119) (r) (hostID, customerID string, isGlobal, ok bool) Bearer auth for agent-facing endpoints (global OR per-host key) Sibling of checkAuthCustomer — do NOT mix the two token namespaces. Global key requires the host row to already exist (see handleHostReport).
(*Handler).handleEvent + allowedEventTypes hub/internal/api/handler.go (~L1115 / ~L1063) POST /api/v1/event The ONLY controller→hub structured-event ingest Unknown event_type → 400 (add to the map FIRST). Accepted severities: info/warning/error/critical (critical since v0.31.0); anything else coerces to "info" — exact-match lowercase ("Critical" coerces). Tests: hub/internal/api/event_test.go.
(*Handler).handleHostReport hub/internal/api/handler.go (~L464) POST /api/v1/host-report Agent heartbeat ingest: denorm + guest upsert Body cap via LimitReader; per-host key enforces host_id match (403 on mismatch); received_at is the dead-man's-switch.
(*Handler).handleConfigRetrieve hub/internal/api/handler.go (~L1484) GET /api/v1/config/{id}, header X-Retrieval-Password Canonical password-gated retrieval endpoint Constant-time compare vs cfg.RetrievalPassword; 404-before-401 ordering. handleArtifactManifest mirrors it EXACTLY — keep them in lockstep. Also the Day-0 claim entry point: calls claimEngine.EnsureIssued + bakes the hash via configgen.Generate(…, claimState).
claim.Engine hub/internal/claim/engine.go EnsureIssued / Resend / RequestReset / MarkClaimed (all take *store.CustomerConfig) Customer-claim code engine (v0.50.0, F-4) Stores bcrypt(code) ONLY — plaintext lives just in the email send. EnsureIssued is idempotent (never rotates/re-sends an existing row). Wired via api.SetClaimEngine + web.SetClaimEngine; the Mailer seam is *notify.Dispatcher.
(*Store).RotateClaimCode / GetClaim / MarkClaimed hub/internal/store/store.go claim-state CRUD customer_claims row (v0.50.0) RotateClaimCode bumps generation (single active code) + PRESERVES claimed_at (reset never un-claims); MarkClaimed is set-only.
configgen.Generate hub/internal/configgen/configgen.go (~L16) (templateYAML string, cfg *store.CustomerConfig, claimState *store.ClaimState) (string, error) Generate a customer's controller.yaml The 3rd arg (nil-safe) bakes web.claim_code_*. The REAL config-retrieve path issues+emails first (EnsureIssued); the preview/DR paths bake read-only via store.GetClaim.
notify.FormatClaimEmail / (*Dispatcher).SendClaimEmail hub/internal/notify/{templates,dispatcher}.go (kind, customerID, email, domain, code) Hungarian claim/reset/claimed emails kind ∈ claim|reset|claimed. The dispatcher method IS the claim.Mailer. Never log the code.
writeJSON hub/internal/api/dr.go (~L25) (w, code int, v any) JSON responses in api package Only used in dr.go so far; prefer it over ad-hoc byte-writes for new endpoints.
(*mailRateLimiter).allow hub/internal/api/mail.go (~L48) (key string) bool Per-key token-bucket rate limiting Refill = perMinute/60 per sec, burst = perMinute; in-memory (lost on restart, accepted). now is injectable for tests.

Alerting / Resend / dispatcher (hub/internal/notify/, monitor/)

Symbol File Short signature Use for Gotchas
(*Dispatcher).ProcessEvent hub/internal/notify/dispatcher.go (~L67) (customerID, eventType, severity, message, detailsJSON, source) THE notification pipeline (operator + customer channels, cooldowns, prefs) Call in a goroutine (handlers do go d.ProcessEvent(...)). No Resend key → silent no-op. eventType=="test" bypasses prefs/cooldown.
severityNotifies hub/internal/notify/dispatcher.go (~L56) (severity string) bool Deciding whether a severity emails warning/error/critical notify; info intentionally doesn't; anything else is logged as unrecognized (v0.24.0 fix — do not regress).
FormatOperatorEmail / FormatCustomerEmail hub/internal/notify/templates.go (~L24 / ~L118) (...) (subject, body) Operator (English) / customer (Hungarian) email bodies Customer messages come from the customerMessages map — add the Hungarian text when adding an event type. Budapest TZ via package init().
monitor.EventNotifyFunc hub/internal/monitor/staleness.go (~L14) func(customerID, eventType, severity, message, detailsJSON, source) Decoupling checkers from notify; wired to dispatcher.ProcessEvent in main May be nil — always nil-check before calling (all checkers do).
(*Store).LogNotification hub/internal/store/store.go (~L433) (customerID, eventType, severity, message, status, errorMsg, channel) Audit trail of every send attempt (sent/failed, per channel) Log BOTH success and failure (dispatcher does).

App-mail passthrough (hub/internal/mailrelay/)

Symbol File Short signature Use for Gotchas
mailrelay.Sender / (*ResendSMTP).Send hub/internal/mailrelay/relay.go (~L24 / ~L50) Send(ctx, raw []byte, mailFrom, rcptTo) error Raw-MIME passthrough to Resend SMTP Deliberately separate from notify's HTTP path — parse-then-API drops inline CID images (spike-proven). Do NOT "unify" them. Delivery verdict lands at DATA-close.
mailrelay.FromDomain hub/internal/mailrelay/relay.go (~L130) (raw []byte) (string, error) From-HEADER domain extraction for allowlisting Header domain, not envelope — Resend checks the header.

Web auth / session / CSRF (hub/internal/web/)

Symbol File Short signature Use for Gotchas
(*Server).RequireAuth hub/internal/web/server.go (~L359) (next http.Handler) http.Handler Session-cookie OR Basic-auth gate for all web routes Empty effective hash disables auth entirely (dev mode). Browsers → /login redirect; JSON-ish requests → 401.
(*Server).effectivePasswordHash hub/internal/web/server.go (~L118) () string THE single source for the operator login hash — call this, never read configPasswordHash Precedence: hub_settings DB override (set via Configuration UI) wins, else the hub.yaml auth.password_hash seed. ConfigMap = break-glass reset. Change it via POST /configuration/password (handleChangePassword).
(*Server).validateCSRF hub/internal/web/server.go (~L446) (r) bool CSRF check — enforced centrally in web.ServeHTTP for every non-GET No session cookie → returns true (Basic-auth path is exempt). New POST routes get CSRF for free; forms MUST embed csrfField.
(*Server).csrfField hub/internal/web/server.go (~L483) (r) template.HTML Hidden _csrf input for HTML forms Pass into template data on every form-rendering handler.
(*Server).CleanupSessions hub/internal/web/server.go (~L110) (ctx) — goroutine Expired-session sweeper Started once from main; 15-min tick.

Status tokens & template funcmap (hub/internal/web/)

Symbol File Short signature Use for Gotchas
funcMap in web.New hub/internal/web/server.go (~L67) template.FuncMap{...} ALL template helpers (timeAgo, statusColor, json, hubVersion…) Add new template funcs HERE, nowhere else. Tested by hub/internal/web/funcmap_test.go + render_test.go.
inline_confirm_js partial (v0.52.0) hub/internal/web/templates/inline_confirm.html {{template "inline_confirm_js"}} in the page <head> Inline "question + Igen/Mégse" confirm for consequential buttons — data-confirm="…" on the button, or felhomConfirm(el, q, onYes) from JS NEVER native confirm()/prompt() — OS-modals freeze browser automation (F-16); scripts/hub_confirm_gate.py enforces zero. Uses requestSubmit so formaction sub-buttons riding a parent form work. NOT for the danger-zone typed-confirm cascade.
timeAgo hub/internal/web/server.go (~L603) (t time.Time) string Human-relative timestamps in UI
statusColor hub/internal/web/server.go (~L630) (status string) string Status → design-system-v2 token (nominal/warn/crit/neutral) Class SUFFIX only, never inline color (D4). Exception-color principle: healthy = blue/neutral.
(*Server).hostStatus + hostStatusClass/hostStatusLabel hub/internal/web/hosts.go (~L16/34/48) (lastReport *time.Time) string Host liveness badge Uses the SAME threshold as HostStalenessChecker (down = 2× stale) — never invent a second definition.
parseSQLiteTime hub/internal/store/store.go (~L1160) (s string) time.Time Parsing ANY timestamp read from SQLite modernc/sqlite returns multiple formats; raw time.Parse will intermittently zero out. Always use this.
compareVersions hub/internal/web/server.go (~L571) (a, b string) int X.Y.Z comparisons in web (floor checks, update-available) Returns 0 on parse error — unparseable compares as "equal" (see §3).

Host views & lifecycle / offsite endpoints (v0.47.0, hub/internal/web + store)

Symbol File Short signature Use for Gotchas
(*Server).hostDetailData hub/internal/web/hosts.go (~L282) (host *store.Host, r) map[string]interface{} The ONE view-model builder for the shared host_detail_body sub-template (standalone /hosts/{id} + customer Host tab) Booleans/counts only for DR/escrow; carries Deletable (= status != "ok") which gates the danger-zone card. Never add a secret field.
host_detail_body sub-template hub/internal/web/templates/host_detail_body.html {{template "host_detail_body" .}} Rendering a host's detail sections on ANY surface One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with {{.HostID}} (the customer page renders N instances).
(*Store).ListHostsByCustomer hub/internal/store/store.go (~L1620) (customerID) ([]Host, error) A customer's hosts, host_id order A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer.
(*Server).configFormData (v0.49.0) hub/internal/web/configs.go (~L430) (r, isNew, cfg, overrides, errMsg) configFormView The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) overrides=nil → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed).
config_form_body sub-template (v0.49.0) hub/internal/web/templates/config_form_body.html {{template "config_form_body" <configFormView>}} Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) The floor/geo/danger cards on the Edit tab are SIBLINGS after </form> — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight <script>.
(*Store).CountHostArtifacts / DeleteHost hub/internal/store/store.go (~L1640/~L1690) (hostID) (HostArtifacts, error) / (hostID, deleteEscrow bool) error Host-delete impact preview + the ONE-transaction cascade ONLINE gate lives in the handler, escrow gate in the store (ErrHostEscrowPresent, tx never starts). log_bundles die by scope_id == host_id ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out.
(*Store).ListWGEndpoints / DeleteWGEndpoint hub/internal/store/wg.go (~L64/~L86) () ([]WGEndpoint, error) / (endpointID) error The /offsite endpoint-management surface GetWGEndpoint (lowest id, LIMIT 1) stays THE allocation/sync endpoint — do not switch allocator/reconciler/desired-state to the list without the wg_peers.endpoint_id migration arc. Peers-in-subnet guards live in hub/internal/web/offsite.go.

Artifact manifest / Day-0 trust root

Symbol File Short signature Use for Gotchas
(*Store).GetArtifactManifest / SetArtifactManifest hub/internal/store/store.go (~L933 / ~L944) () ArtifactManifest / (m) error The DB-backed (hub_settings) Day-0 artifact record This is the checksum TRUST ROOT the host-bootstrap verifies against — distinct from Gitea, which only stores bytes.
(*Store).GetOperatorPasswordHash / SetOperatorPasswordHash hub/internal/store/store.go (~L1350) () string / (hash) error The DB-backed (hub_settings) operator login password override Read via Server.effectivePasswordHash(), not directly. "" = no override (config seed authoritative). Store the bcrypt hash, never the plaintext.
(*Server).handleSetArtifacts + resolveArtifactSHA hub/internal/web/configs.go (~L644 / ~L680) POST /configuration/artifacts Operator UI to vouch artifact versions With a Gitea client the sha is fetched AUTHORITATIVELY (submitted sha ignored); fetch failure refuses the save. Manual sha only in the no-creds fallback.
(*gitea.Client).ListVersions / FileSHA256 hub/internal/gitea/gitea.go (~L47 / ~L72) (ctx, pkg) ([]string, error) / (ctx, pkg, ver, file) Read-only Gitea generic-package metadata sha comes from package metadata — artifact bytes are never downloaded. Newest-semver-first sort.
(*Server).artifactChoices hub/internal/web/server.go (~L155) (ctx, pkg, file) []artifactChoice Version+sha dropdown data nil Gitea client / unreachable → nil → UI degrades to manual entry. One bad version drops itself, not the list.
(*Handler).handleArtifactManifest hub/internal/api/handler.go (~L1550) GET /api/v1/artifacts/{id} Serving the vouched set to the bootstrap script Auth mirrors handleConfigRetrieve exactly. Unset manifest = empty fields, not an error.
normalizeFloorInput / normalizeSHA256 hub/internal/web/configs.go (~L27 / ~L627) (raw string) (string, bool) Validating operator-typed versions / shas Empty string is VALID (means "clear"). Reuse for any new version/sha form field.

PBS DR tier / tenantsync (hub/internal/tenantsync/, hub/internal/web/pbsdr.go, hub/internal/api/pbsdr.go)

Symbol File Short signature Use for Gotchas
tenantsync.Client (Provision/Reissue/Fingerprint) hub/internal/tenantsync/client.go (ctx, customerID) (*Result, error) ep0 per-customer PBS tenancy over the pinned-SSH forced-command channel (the wgsync twin) Result.TokenSecret is transient custody → SaveHostPBSSecret immediately, never log the struct. Error paths NEVER embed stdout (the secret channel) — do not "improve" diagnostics by quoting the response. ErrTokenExists is typed: provision refuses an existing token; re-issue is the explicit path.
(*Store).SaveHostPBSSecret / ConsumeHostPBSSecret hub/internal/store/pbsdr.go (hostID, value) / (hostID) (string, error) HOST-scoped consume-once secret (the one_time_secrets host twin) Same-tx mark-consumed; re-save resets consumption (re-issue supersedes). The agent consumes via POST /api/v1/hosts/{id}/pbs/consume-token (hub/internal/api/pbsdr.go).
(*Server).applyPBSDR + mergePBSDR/readPBSDR hub/internal/web/pbsdr.go (ctx, r, cfg) error The config form's DR-tier section → HOST desired_json pbs_dr descriptor + generation bump Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by cfg.DRTier (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy.
(*Server).pbsdrProvisionAtom + PBSDRAutoProvision hub/internal/web/pbsdr.go (ctx, customerID, host, storageID) (blocked string, err error) / (ctx, customerID) The shared fresh-provision cascade atom; the WG-registration hook target (api SetWGRegisteredHook, wired in hub/cmd/hub/main.go when tenantsync is on) blocked != "" = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration.
cfg.DRTier + offsite coupling hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) bool Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field dr_tier (formBool helper).
pbsdrheal.Reconciler + NewActions hub/internal/pbsdrheal/reconciler.go NewReconciler(st, act, logger) · RestrictToHost(hostID) · Run(ctx)/Trigger() PBS-DR SELF-HEAL: re-arms a consumable secret for a box stuck in waiting_secret/consumed_failed after losing its converged marker (re-install/rollback). From SPIKE-pbsdr-selfheal-2026-07-15. Primary heal = re-stage the stored secret (no ep0 call, NO generation bump — a bump = agent refetch loop). Escalate to Re-issue only when no stored secret / consumed_failed. NEVER re-run pbsdrProvisionAtom (refuses ErrTokenExists) or blind-timer Re-issue (hash/gen thrash). Converged/disabled/verify_failed/DR-OFF = no-op. Debounce ≥2 distinct reports. PBSDRHEAL_ONLY_HOST scopes a supervised rollout. Fake seam: fakeActions in reconciler_test.go.
(*Store).RestageHostPBSSecret + PBSDRHealStates hub/internal/store/pbsdr.go (hostID) (restaged bool, err) / () ([]PBSDRHealRow, error) The self-heal store primitives: clear consumed_at IFF a row exists (re-arm the SAME value); the fleet work-set query (descriptor enable/provision + latest report pbs_dr.state + id) RestageHostPBSSecret does NO insert, NO value change, NO generation bump (restaged=false = no row → caller escalates). PBSDRHealStates mirrors GetHostOOBStates' latest-report-per-host join; malformed JSON → zero values, never an error.
(*Server).ReissuePBSDR hub/internal/web/pbsdr.go (ctx, customerID) error The non-HTTP core of the operator Re-issue button — the self-heal reconciler's escalation seam (satisfies pbsdrheal.Reissuer) Reuses tenantsync.Reissue + SaveHostPBSSecret + descriptor bump — NOT a re-run of pbsdrProvisionAtom. Keep in lockstep with the tail of handlePBSDRReissue (which is unchanged; the operator button's 303/400 codes are pinned by tests).
parseHostCapabilities + capabilityView hub/internal/web/hosts.go (reportJSON) []capabilityView Host-page capability chips (ok/degraded/inactive) inactive (agent v0.86.0) = badge-neutral, NEVER warn/error — disabled ≠ degraded; unknown future statuses fall to the degraded styling (surface, don't hide). capabilitiesNeedDRMigration keys the pre-v1.15.0 migration hint on pbsdr-* + "binary not found".

Config generation & secrets hygiene (hub/internal/configgen/)

Symbol File Short signature Use for Gotchas
configgen.Generate hub/internal/configgen/configgen.go (~L16) (templateYAML, cfg) (string, error) Producing a customer controller.yaml Programmatic overrides (customer id/hub url/api_key) ALWAYS win over config_json; fresh session secret per generation.
configgen.RandomHex hub/internal/configgen/configgen.go (~L110) (n int) (string, error) crypto/rand hex tokens (API keys, session secrets)
configgen.RandomPassphrase hub/internal/configgen/passphrase.go (~L35) (wordCount int) (string, error) Human-dictatable Hungarian passphrases (retrieval passwords) ~29K-word embedded list; 5 words ≈ 74 bits.
(*Store).EffectiveMinControllerVersion hub/internal/store/store.go (~L960) (customerID) string Resolving the floor that actually applies (per-customer → global) "" = no floor (Phase 2 inert).

Assets, store misc, scheduling

Symbol File Short signature Use for Gotchas
assets.Manager (New, ServeFile, ReSeed) hub/internal/assets/assets.go (~L44/190/122) seed-dir → PVC sync + manifest Serving app logos/screenshots to controllers ServeFile sanitizes to filepath.Base (no traversal). isAssetFile enforces the naming convention.
copyFile hub/internal/assets/assets.go (~L259) (src, dst) error THE atomic file write (tmp + rename) in the hub Copy this shape for any new on-disk write.
fileSHA256 hub/internal/assets/assets.go (~L244) (path) (string, error) Streaming sha256 of a file
(*Store).SaveEvent hub/internal/store/store.go (~L1003) (...) (int64, error) Persisting ANY event (controller or hub source) Pair with dispatcher/onEvent — saving alone never notifies.
(*Store).RequestLogTail / GetPendingLogTailRequests / SaveAppLogTail hub/internal/store/logtail.go pending-intent + consume-once fulfillment THE ACK-flag pull pattern for hub→box requests (copy for any new one) SaveAppLogTail clears the request in the SAME tx (consume-once) + prunes to last 2 per (customer,app); the hub NEVER connects into a box
(*Store).RequestLogBundle / PendingLogBundleRequest / SaveLogBundle / PurgeExpiredLogBundles hub/internal/store/logbundle.go component (controller/agent) log pulls — the v0.46.0 sibling of logtail.go box-component debug-ring pulls; gzip custody, newest-3, 72 h TTL on the 60 s sweep scope = customer_id (controller/report ACK) vs host_id (agent/heartbeat envelope); SaveLogBundle runs the SECRET GATE fail-closed (blocked flag row, no payload) and clears the request in the same tx; [REDACTED]/checksums pass by design
upsertAppIssue dismissal/context semantics hub/internal/store/telemetry.go ON CONFLICT CASE guards Issue dismissal + first-capture-wins context Un-dismiss ONLY on excluded.last_seen > dismissed_at; context adopted only while stored one is empty — do not "simplify" either CASE (red-proofed)
store.GuestID hub/internal/store/store.go (~L1268) (hostID string, vmid int) string Canonical guest primary key Never hand-concatenate host+vmid.
scheduleDaily hub/cmd/hub/main.go (~L449) (ctx, name, "HH:MM", fn, logger) Daily jobs in Europe/Budapest (prune etc.) Blocking — run as goroutine. parseHM returns 0,0 (midnight) on bad input.

2. Canonical patterns (copy structure from THE named file)

Pattern Canonical file Key traits
Monitor checker hub/internal/monitor/staleness.go Seed state on construction WITHOUT emitting events; in-memory states map under mutex; periodic Check(); emitTransition = SaveEvent then nil-checked onEvent; cleanup of vanished IDs. HostStaleness/HostDisk/HostLeaf/HostCapability/StorageFill all follow it.
API endpoint hub/internal/api/handler.go ServeHTTP (~L139) + any handler Path switch in ServeHTTP; first line of every handler = checkAuth{Customer,Host}; io.LimitReader body cap; typed anonymous payload struct; explicit 4xx strings.
Web POST action hub/internal/web/configs.go handleSetGlobalFloor (~L602) CSRF enforced centrally in web ServeHTTP; validate via normalize* helper; POST-redirect-GET with ?flash= token; log INFO on success.
Optional dependency injection hub/internal/web/server.go Set* setters (~L131148) / api handler SetDispatcher etc. Constructor takes hard deps; optional subsystems wired via SetX after construction; nil = graceful degradation (never panic).
Seam-injected sender for tests hub/internal/notify/dispatcher.go sendEmailFn (~L33) Function-field defaulting to the real impl in the constructor; tests overwrite it. Same idea: mailRateLimiter.now, mailrelay.Sender fake.
Website page website/index.html UTF-8 with BOM; shared <nav>/<footer> byte-identical across pages (only class="active" differs); two-tone H1 = <h1>…<span>accent</span></h1>; all styling in website/assets/site.css tokens (:root) — zero embedded <style>; ?v=N cache-bust on site.css/icons.svg; umami snippet; no CDN fonts; no emoji (sprite icons.svg instead).
Gate script scripts/site_gates.py Byte-level mechanical gates (BOM, emoji codepoint ranges, nav/footer diff, analytics, banned tokens, cache-bust); run python scripts/site_gates.py after ANY website change; non-zero exit on failure.
Fetch-validate-install (shell) scripts/felhom-host-install.sh step_agent_install (~L1108) fetch_raw to mktemp → syntax-check (bash -n) → install -m0755 -o root -g root → only then activate; guarded-mkfs wrapper installed BEFORE the sudoers that references it (ordering is the safety property). All mutations through run() (dry-run aware).
Install-profile gate (shell) scripts/felhom-host-install.sh --mode appliance|byo (GL-2, v1.10.0) Mandatory-flag profile (no default), refusals at argv time BEFORE any prompt/step, risky step gated at its CALL SITE (one auditable place — never a branch inside the step), mode persisted to state.json + resume-mismatch refusal, FELHOM_INSTALL_STATE_DIR override for harness isolation. Harness: scripts/hostinstall-mode-harness.sh (static refusal matrix + grep-invariants + PVE dry-transcript tier; red-proofs run against a mutated scratch copy).
Disclosure↔uninstall parity (shell) scripts/felhom-host-install.sh _uninstall_statement + harness GL4-D (v1.11.0) Every host artifact the byo disclosure names must be removed OR explicitly listed KEPT by run_uninstall; the harness greps the parity (token list). New install-time artifact ⇒ add its removal + disclosure line + parity token in the SAME commit. Drive data rule: plain umount only, never -l/-f, never any format op under /mnt/felhom-drives.
Website deploy (manifest) manifests/webpage.yaml git-sync sidecar (sparse-checkout /website/ + /scripts/, --link=current) + init container waits for first sync; nginx serves current/website; push to main = deployed, no image build.
Secret handling (manifest) manifests/hub.yaml (env, ~L142) Secrets via secretKeyRef to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). report-api (the operator bearer, v0.53.0) is deliberately NOT optional: — a missing Secret fails Ready instead of booting an unauthenticatable hub. scripts/manifest_bearer_gate.py (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. ERRATA (2026-07-03): gitea-creds is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix).
Hub deploy (GitOps) manifests/hub.yaml image: (~L129) Pinned explicit tag, bumped in git, deliberate ArgoCD sync (auto-sync OFF). Code push alone deploys nothing.

3. Dangerous lookalikes — do NOT reuse

Trap Why it bites Use instead
(*Handler).handleNotify + formatNotificationEmail + sendResendEmail (hub/internal/api/handler.go ~L1289/1624/1589) Legacy pre-dispatcher notification trio: no cooldowns, no operator channel, no allowedEventTypes gate, duplicate Hungarian formatter. Controller path is FROZEN until slice-10 cutover. POST /api/v1/eventDispatcher.ProcessEvent + notify.Format*Email
Severity "critical" POSTed to a PRE-v0.31.0 hub Fixed in hub v0.31.0 (handleEvent now accepts critical). Older hubs coerce critical"info", which never notifies — silent alert loss. Case-variants ("Critical") still coerce on every version. Against an old hub send warning/error; otherwise lowercase critical is safe
compareVersions for anything security-ish (hub/internal/web/server.go ~L571) Returns 0 (equal) on unparseable input — a garbage version passes a floor check. gitea.compareSemver behaves differently (lexical fallback). Validate input with normalizeFloorInput first; then compareVersions is safe
Inline stringData secrets à la manifests/felhom.secret.yaml Commits real credentials to git (healthchecks superuser pw, umami APP_SECRET/POSTGRES_PASSWORD, gitea-creds admin password still live there). Out-of-band kubectl create secret + secretKeyRef (hub.yaml resend-api pattern; runbook documentation/runbooks/secrets.md)
kubectl apply / kubectl set image on manifests/ ArgoCD app felhom reverts drift on next sync; live state lies about git. Edit manifest in git → push → ArgoCD sync (CLAUDE.md steps 35)
:latest image tag in manifests Re-push doesn't change the manifest → no redeploy; Synced/Rollback misreport. Pinned version tag, bumped per deploy
grep/regex hunting emoji in website HTML Windows grep false-negatives multibyte emoji (proven in D0). python scripts/site_gates.py (codepoint-range check)
Adding a website page without touching site_gates.py PAGES list (scripts/site_gates.py ~L22) is explicit — an unlisted page is silently ungated (BOM/nav/emoji drift undetected). Add the filename to PAGES in the same commit

4. Seams & interfaces (testing + cross-repo)

Interface Defined in Implemented by Fakes/tests at
mailrelay.Sender hub/internal/mailrelay/relay.go (~L24) ResendSMTP (prod) fake sender in hub/internal/api/mail_test.go; hub/internal/mailrelay/relay_test.go
Dispatcher.sendEmailFn (func seam) hub/internal/notify/dispatcher.go (~L33) (*Dispatcher).sendEmail (Resend HTTP) hub/internal/notify/dispatcher_test.go
monitor.EventNotifyFunc hub/internal/monitor/staleness.go (~L14) closure over Dispatcher.ProcessEvent (main.go) hub/internal/monitor tests (captured-events func)
api.ConfigTemplateProvider hub/internal/api/handler.go (~L24) web.TemplateFetcher (Gitea-pulled controller.yaml template) stub providers in api tests
api.LatestVersionProvider hub/internal/api/handler.go (~L31) web.VersionChecker (registry poll) hub/internal/api/config_version_ack_test.go
mailRateLimiter.now (func seam) hub/internal/api/mail.go (~L27) time.Now hub/internal/api/mail_test.go clock injection
web.tenancyProvisioner hub/internal/web/pbsdr.go *tenantsync.Client (pinned SSH to ep0's felhom-tenantsync) fakeTenancy in hub/internal/web/pbsdr_test.go; in-process SSH server in hub/internal/tenantsync/client_test.go
Cross-repo: ep0 tenancy surface scripts/felhom-tenantsync.sh (JSON stdin/stdout forced command) installed on ep0 per runbook offsite-endpoint.md §10 provision/reissue/fingerprint ops; token secret rides stdout ONLY; the peersync script/key are untouched
Cross-repo: controller → hub POST /api/v1/report (frozen) + POST /api/v1/event felhom-controller repo new event types MUST enter allowedEventTypes (hub/internal/api/handler.go ~L1063) or the controller gets 400
Cross-repo: agent → hub POST /api/v1/host-report, /host-enroll, jobs/desired-state/escrow routes (handler.go ~L145189) felhom-agent repo hub/internal/api/host_test.go, desired_test.go, escrow_test.go, dr_test.go
Cross-repo: Day-0 bootstrap → hub GET /api/v1/config/{id} + /artifacts/{id} (X-Retrieval-Password) scripts/felhom-host-install.sh (fetches + sha256-verifies against the hub-vouched manifest) hub/internal/api/artifact_test.go
Cross-repo: controller ← hub assets GET /api/v1/assets/manifest + /assets/file/{name} felhom-controller pulls app logos/screenshots assets manifest sha-based change detection

5. Extension points (where new features plug in)

  • New event type: add to allowedEventTypes (hub/internal/api/handler.go ~L1063) and customerMessages (hub/internal/notify/templates.go) and the customer-prefs default list if customer-notifiable. Missing the first = controller POST 400s (the known gotcha).
  • New monitor checker: copy hub/internal/monitor/staleness.go (§2 pattern); wire in hub/cmd/hub/main.go with an EventNotifyFunc; severity must be warning/error/critical to notify.
  • New API route: switch in api.ServeHTTP (handler.go ~L139); auth helper first line.
  • New web page/action: switch in web.ServeHTTP (server.go ~L182) — non-GET gets CSRF automatically; template into hub/internal/web/templates/ (embedded FS, parsed in web.New); new helpers into the funcMap (server.go ~L67).
  • New template func: funcMap in web.New only; add a case to hub/internal/web/funcmap_test.go.
  • New daily job: scheduleDaily in hub/cmd/hub/main.go + add pruning to pruneAll if data grows.
  • New site gate: append to scripts/site_gates.py; new website pages go into its PAGES list.
  • New artifact kind (Day-0): consts pkg*/file* (hub/internal/web/server.go ~L27), ArtifactManifest fields + settings keys (hub/internal/store/store.go ~L905), handleSetArtifacts, artifactManifestResponse (handler.go), and the install script's verify step.
  • New host-install step: step_* function in scripts/felhom-host-install.sh using run()/fetch_raw/die helpers; keep dry-run coverage.

6. Known duplication (observed — NOT fixed)

  • Resend HTTP sender ×2: (*Handler).sendResendEmail (hub/internal/api/handler.go ~L1589) ≈ (*Dispatcher).sendEmail (hub/internal/notify/dispatcher.go ~L185) — byte-near-identical POST to api.resend.com. Kept because the handler copy serves the frozen legacy /notify path.
  • Hungarian customer-email formatter ×2: formatNotificationEmail (hub/internal/api/handler.go ~L1624) vs notify.FormatCustomerEmail (hub/internal/notify/templates.go ~L118). Legacy vs dispatcher; the legacy one lacks the per-event-type message map.
  • Semver compare ×2 with DIFFERENT fallback semantics: web.compareVersions (hub/internal/web/server.go ~L571, parse error → 0) vs gitea.compareSemver (hub/internal/gitea/gitea.go ~L115, parse error → lexical). Documented as deliberate (import-cycle avoidance) in gitea.go, but the behavior drift is not.
  • Checker-family structural repetition: staleness.go vs host_staleness.go, and host_disk.go vs storage_fill.go (band/bandRank/bandLabel vs bandForPercent) — same skeleton re-implemented per domain; treated as the accepted §2 pattern rather than a defect.
  • Duration formatting ×2: monitor.formatDuration (hub/internal/monitor/staleness.go ~L187) vs web.timeAgo (hub/internal/web/server.go ~L603) — different audiences (email vs UI) but overlapping logic.