49 KiB
49 KiB
REUSE.md — felhom.eu (hub + website + scripts)
Before writing new code, check here. Canonical helpers, patterns to copy, traps to avoid. Maintenance: update in the SAME commit that adds/changes/deprecates a shared helper. Entries cite file + symbol. Line numbers are landmarks only — reconfirm before editing.
1. Canonical helpers (MUST reuse — do not reinvent)
Report ingest & API auth (hub/internal/api/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Handler).checkAuthCustomer |
hub/internal/api/handler.go (~L94) | (r) (customerID string, isGlobal, ok bool) |
Bearer auth for controller-facing endpoints (global key OR per-customer key) | Global key → ("", true, true): caller must then trust body customer_id. Constant-time compare on global key. |
(*Handler).checkAuthHost |
hub/internal/api/handler.go (~L119) | (r) (hostID, customerID string, isGlobal, ok bool) |
Bearer auth for agent-facing endpoints (global OR per-host key) | Sibling of checkAuthCustomer — do NOT mix the two token namespaces. Global key requires the host row to already exist (see handleHostReport). |
(*Handler).handleEvent + allowedEventTypes |
hub/internal/api/handler.go (~L1115 / ~L1063) | POST /api/v1/event |
The ONLY controller→hub structured-event ingest | Unknown event_type → 400 (add to the map FIRST). Accepted severities: info/warning/error/critical (critical since v0.31.0); anything else coerces to "info" — exact-match lowercase ("Critical" coerces). Tests: hub/internal/api/event_test.go. |
(*Handler).handleHostReport |
hub/internal/api/handler.go (~L464) | POST /api/v1/host-report |
Agent heartbeat ingest: denorm + guest upsert | Body cap via LimitReader; per-host key enforces host_id match (403 on mismatch); received_at is the dead-man's-switch. |
(*Handler).handleConfigRetrieve |
hub/internal/api/handler.go (~L1484) | GET /api/v1/config/{id}, header X-Retrieval-Password |
Canonical password-gated retrieval endpoint | Constant-time compare vs cfg.RetrievalPassword; 404-before-401 ordering. handleArtifactManifest mirrors it EXACTLY — keep them in lockstep. Also the Day-0 claim entry point: calls claimEngine.EnsureIssued + bakes the hash via configgen.Generate(…, claimState). |
claim.Engine |
hub/internal/claim/engine.go | EnsureIssued / Resend / RequestReset / MarkClaimed (all take *store.CustomerConfig) |
Customer-claim code engine (v0.50.0, F-4) | Stores bcrypt(code) ONLY — plaintext lives just in the email send. EnsureIssued is idempotent (never rotates/re-sends an existing row). Wired via api.SetClaimEngine + web.SetClaimEngine; the Mailer seam is *notify.Dispatcher. |
(*Store).RotateClaimCode / GetClaim / MarkClaimed |
hub/internal/store/store.go | claim-state CRUD | customer_claims row (v0.50.0) |
RotateClaimCode bumps generation (single active code) + PRESERVES claimed_at (reset never un-claims); MarkClaimed is set-only. |
configgen.Generate |
hub/internal/configgen/configgen.go (~L16) | (templateYAML string, cfg *store.CustomerConfig, claimState *store.ClaimState) (string, error) |
Generate a customer's controller.yaml | The 3rd arg (nil-safe) bakes web.claim_code_*. The REAL config-retrieve path issues+emails first (EnsureIssued); the preview/DR paths bake read-only via store.GetClaim. |
notify.FormatClaimEmail / (*Dispatcher).SendClaimEmail |
hub/internal/notify/{templates,dispatcher}.go | (kind, customerID, email, domain, code) |
Hungarian claim/reset/claimed emails | kind ∈ claim|reset|claimed. The dispatcher method IS the claim.Mailer. Never log the code. |
writeJSON |
hub/internal/api/dr.go (~L25) | (w, code int, v any) |
JSON responses in api package | Only used in dr.go so far; prefer it over ad-hoc byte-writes for new endpoints. |
(*mailRateLimiter).allow |
hub/internal/api/mail.go (~L48) | (key string) bool |
Per-key token-bucket rate limiting | Refill = perMinute/60 per sec, burst = perMinute; in-memory (lost on restart, accepted). now is injectable for tests. |
Alerting / Resend / dispatcher (hub/internal/notify/, monitor/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Dispatcher).ProcessEvent |
hub/internal/notify/dispatcher.go (~L88) | (customerID, eventType, severity, message, detailsJSON, source) |
THE notification pipeline (operator + customer channels, cooldowns, prefs) | Call in a goroutine (handlers do go d.ProcessEvent(...)). No Resend key → silent no-op. eventType=="test" bypasses prefs/cooldown (and since v0.71.0 also mails the operator). *_recovered routes via the explicit recovery branch BEFORE the severity gate (v0.71.0) — see processRecovery. |
(*Dispatcher).processRecovery |
hub/internal/notify/dispatcher.go (~L160) | internal | node_recovered/host_recovered routing (audit F11) |
Operator always (1 h cooldown); customer iff PAIRED — a customer-channel sent stale/down row newer than the last sent recovery (store.LastCustomerSentAt); enabled_events deliberately ignored for recovery; timestamp ties → no mail (flap-safe). Severity stays info — never "fix" that by widening severityNotifies. |
severityNotifies |
hub/internal/notify/dispatcher.go (~L77) | (severity string) bool |
Deciding whether a severity emails | warning/error/critical notify; info intentionally doesn't; anything else is logged as unrecognized (v0.24.0 fix — do not regress). Recovery mails exist DESPITE this gate (eventType branch), not through it. |
priorityHeaders |
hub/internal/notify/dispatcher.go (~L56) | (severity string) map[string]string |
High-priority mail-client nudge (audit F14-light) | error/critical → X-Priority: 1 + Importance: high; everything else nil — a warning/info mail must NOT masquerade as urgent (red-proofed). |
sendEmailFn seam / sendEmail |
hub/internal/notify/dispatcher.go (~L33 / ~L300) | func(to, subject, textBody string, headers map[string]string) error |
Test seam for all sends; Resend POST | Signature grew a headers param in v0.71.0 — payload carries "headers" only when non-empty. Tests capture recipient+subject+headers through the seam. |
FormatOperatorEmail / FormatCustomerEmail |
hub/internal/notify/templates.go (~L24 / ~L118) | (...) (subject, body) |
Operator (English) / customer (Hungarian) email bodies | Customer messages come from the customerMessages map — add the Hungarian text when adding an event type. Budapest TZ via package init(). Operator icon is eventType-aware: *_recovered → ✅ (severity is the fallback). |
monitor.EventNotifyFunc |
hub/internal/monitor/staleness.go (~L14) | func(customerID, eventType, severity, message, detailsJSON, source) |
Decoupling checkers from notify; wired to dispatcher.ProcessEvent in main |
May be nil — always nil-check before calling (all checkers do). |
(*Store).LogNotification |
hub/internal/store/store.go (~L433) | (customerID, eventType, severity, message, status, errorMsg, channel) |
Audit trail of every send attempt (sent/failed, per channel) | Log BOTH success and failure (dispatcher does). Since v0.71.0 these rows are also the recovery PAIRING evidence — never prune them casually. |
(*Store).LastCustomerSentAt |
hub/internal/store/store.go (~L815) | (customerID, eventTypes []string) (time.Time, bool, error) |
Pairing-evidence query (max customer-channel sent created_at over types) |
Uses the (customer_id, created_at DESC) index. Empty type list → (zero, false, nil). |
(*Store).SeedNotificationPrefs |
hub/internal/store/store.go (~L850) | (customerID, email, enabledEvents) (seeded bool, err) |
Claim-time prefs seeding (audit F12) | INSERT OR IGNORE — never an upsert (red-proofed); empty email = no-op. Customer edits go through SaveNotificationPrefs, seeds NEVER do. |
App-mail passthrough (hub/internal/mailrelay/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
mailrelay.Sender / (*ResendSMTP).Send |
hub/internal/mailrelay/relay.go (~L24 / ~L50) | Send(ctx, raw []byte, mailFrom, rcptTo) error |
Raw-MIME passthrough to Resend SMTP | Deliberately separate from notify's HTTP path — parse-then-API drops inline CID images (spike-proven). Do NOT "unify" them. Delivery verdict lands at DATA-close. |
mailrelay.FromDomain |
hub/internal/mailrelay/relay.go (~L130) | (raw []byte) (string, error) |
From-HEADER domain extraction for allowlisting | Header domain, not envelope — Resend checks the header. |
Web auth / session / CSRF (hub/internal/web/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Server).RequireAuth |
hub/internal/web/server.go (~L359) | (next http.Handler) http.Handler |
Session-cookie OR Basic-auth gate for all web routes | Empty effective hash disables auth entirely (dev mode). Browsers → /login redirect; JSON-ish requests → 401. |
(*Server).effectivePasswordHash |
hub/internal/web/server.go (~L118) | () string |
THE single source for the operator login hash — call this, never read configPasswordHash |
Precedence: hub_settings DB override (set via Configuration UI) wins, else the hub.yaml auth.password_hash seed. ConfigMap = break-glass reset. Change it via POST /configuration/password (handleChangePassword). |
(*Server).validateCSRF |
hub/internal/web/server.go (~L446) | (r) bool |
CSRF check — enforced centrally in web.ServeHTTP for every non-GET |
No session cookie → returns true (Basic-auth path is exempt). New POST routes get CSRF for free; forms MUST embed csrfField. |
(*Server).csrfField |
hub/internal/web/server.go (~L483) | (r) template.HTML |
Hidden _csrf input for HTML forms |
Pass into template data on every form-rendering handler. |
(*Server).CleanupSessions |
hub/internal/web/server.go (~L110) | (ctx) — goroutine |
Expired-session sweeper | Started once from main; 15-min tick. |
Status tokens & template funcmap (hub/internal/web/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
funcMap in web.New |
hub/internal/web/server.go (~L67) | template.FuncMap{...} |
ALL template helpers (timeAgo, statusColor, json, hubVersion…) |
Add new template funcs HERE, nowhere else. Tested by hub/internal/web/funcmap_test.go + render_test.go. |
inline_confirm_js partial (v0.52.0) |
hub/internal/web/templates/inline_confirm.html | {{template "inline_confirm_js"}} in the page <head> |
Inline "question + Igen/Mégse" confirm for consequential buttons — data-confirm="…" on the button, or felhomConfirm(el, q, onYes) from JS |
NEVER native confirm()/prompt() — OS-modals freeze browser automation (F-16). scripts/hub_confirm_gate.py asserts zero, but is not wired to run (R-29) — so the rule holds only as long as you keep it. Uses requestSubmit so formaction sub-buttons riding a parent form work. NOT for the danger-zone typed-confirm cascade. |
timeAgo |
hub/internal/web/server.go (~L603) | (t time.Time) string |
Human-relative timestamps in UI | — |
statusColor |
hub/internal/web/server.go (~L630) | (status string) string |
Status → design-system-v2 token (nominal/warn/crit/neutral) | Class SUFFIX only, never inline color (D4). Exception-color principle: healthy = blue/neutral. |
(*Server).hostStatus + hostStatusClass/hostStatusLabel |
hub/internal/web/hosts.go (~L16/34/48) | (lastReport *time.Time) string |
Host liveness badge | Uses the SAME threshold as HostStalenessChecker (down = 2× stale) — never invent a second definition. |
parseSQLiteTime |
hub/internal/store/store.go (~L1160) | (s string) time.Time |
Parsing ANY timestamp read from SQLite | modernc/sqlite returns multiple formats; raw time.Parse will intermittently zero out. Always use this. |
compareVersions |
hub/internal/web/server.go (~L571) | (a, b string) int |
X.Y.Z comparisons in web (floor checks, update-available) | Returns 0 on parse error — unparseable compares as "equal" (see §3). |
Host views & lifecycle / offsite endpoints (v0.47.0, hub/internal/web + store)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Server).hostDetailData |
hub/internal/web/hosts.go (~L282) | (host *store.Host, r) map[string]interface{} |
The ONE view-model builder for the shared host_detail_body sub-template (standalone /hosts/{id} + customer Host tab) |
Booleans/counts only for DR/escrow; carries Deletable (= status != "ok") which gates the danger-zone card. Never add a secret field. |
parseHostAddresses + (*Server).hostNetwork / hostNetworkView (v0.85.0) |
hub/internal/web/hosts.go | (reportJSON) []hostAddressView · (host, reportJSON) hostNetworkView |
The host page's Network card: every routable address the box holds + its WireGuard allocation | Needs agent >= 0.119.0 (minAgentForAddresses); below it the wire has no addresses key and the card renders UNKNOWN, never "no addresses" — an absent signal is not a negative result. WireGuard is TWO facts: the hub's allocation (GetWGPeerForHost, authoritative) AND whether the box confirms holding it — the allocation alone cannot distinguish a live tunnel from a peer that was never applied. The WG row is split out by comparing against the ALLOCATION, never by matching the interface name wg-felhom, which is a unit name that can change. |
(*Store).GetHostRecoveryMeta + (*Server).handleHostRevealRecoveryCredential |
hub/internal/store/host_recovery.go · hub/internal/web/hosts.go | (hostID) (*HostRecoveryMeta, error) · POST /hosts/{id}/reveal-recovery-credential |
The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) | Use GetHostRecoveryMeta on any page-render path — its struct and its SELECT both omit the secret column, so it cannot leak one; GetHostRecoveryCredential (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE recovery_credential_revealed event via SaveEvent and calls NO dispatcher (the handleRequestLogTail shape). api/handler.go handleAdminGetRecoveryCredential (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133. |
host_detail_body sub-template |
hub/internal/web/templates/host_detail_body.html | {{template "host_detail_body" .}} |
Rendering a host's detail sections on ANY surface | One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with {{.HostID}} (the customer page renders N instances). |
(*Store).ListHostsByCustomer |
hub/internal/store/store.go (~L1620) | (customerID) ([]Host, error) |
A customer's hosts, host_id order | A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer. |
(*Server).configFormData (v0.49.0) |
hub/internal/web/configs.go (~L430) | (r, isNew, cfg, overrides, errMsg) configFormView |
The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | overrides=nil → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
config_form_body sub-template (v0.49.0) |
hub/internal/web/templates/config_form_body.html | {{template "config_form_body" <configFormView>}} |
Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after </form> — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight <script>. |
(*Store).CountHostArtifacts / DeleteHost |
hub/internal/store/store.go (~L1640/~L1690) | (hostID) (HostArtifacts, error) / (hostID, deleteEscrow bool) error |
Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (ErrHostEscrowPresent, tx never starts). log_bundles die by scope_id == host_id ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
(*Server).commitCustomerReset (v0.69.0) |
hub/internal/web/customer_reset.go (~L165) | (ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError |
THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the customer_resets journal |
Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). purgeEscrow governs ONLY whether PurgeCustomerResetDBState destroys retained custody: standalone RESET passes the operator's escrow_ack; the DELETE cascade passes false so custody dies exactly once, in its leg 3. Returns a resetLegError carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
(*Store).CustomerResidue / PurgeCustomerResidue (v0.70.0) |
hub/internal/store/customer_delete.go | (customerID) (*CustomerResidue, error) / (customerID) error |
Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | GetCustomers() is REPORT-derived — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared residueQueries list so a table can never be counted-but-not-purged. Includes appliance_registrations + selfbind_tokens (credential-bearing, not telemetry). NEVER touches events, notification_log, host_deletions, customer_resets. |
(*Server).handleCustomerDelete / handleCustomerDeletePreview (v0.69.0) |
hub/internal/web/customer_delete.go | (w, r, customerID) |
THE customer offboarding entry: the guided full-teardown cascade hosts → RESET → purge (R-25b). GET = live inventory JSON for the dialog, POST = the cascade |
There is NO shallow delete path any more — the old handleConfigDelete is gone; do not reintroduce one. Every gate (3 acks, typed customer-id, stale host-count, ONLINE-host refusal) runs BEFORE any write, so a refusal has zero side effects. Leg order is load-bearing twice over: ruling 3 (RESET never sees a host row) and custody purged exactly once, in leg 3. A failed leg retains the journal — a re-run resumes and must pass every gate again. |
(*Store).ListWGEndpoints / DeleteWGEndpoint |
hub/internal/store/wg.go (~L64/~L86) | () ([]WGEndpoint, error) / (endpointID) error |
The /offsite endpoint-management surface | GetWGEndpoint (lowest id, LIMIT 1) stays THE allocation/sync endpoint — do not switch allocator/reconciler/desired-state to the list without the wg_peers.endpoint_id migration arc. Peers-in-subnet guards live in hub/internal/web/offsite.go. |
Artifact manifest / Day-0 trust root
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Store).GetArtifactManifest / SetArtifactManifest |
hub/internal/store/store.go (~L933 / ~L944) | () ArtifactManifest / (m) error |
The DB-backed (hub_settings) Day-0 artifact record | This is the checksum TRUST ROOT the host-bootstrap verifies against — distinct from Gitea, which only stores bytes. |
(*Store).GetOperatorPasswordHash / SetOperatorPasswordHash |
hub/internal/store/store.go (~L1350) | () string / (hash) error |
The DB-backed (hub_settings) operator login password override | Read via Server.effectivePasswordHash(), not directly. "" = no override (config seed authoritative). Store the bcrypt hash, never the plaintext. |
(*Server).handleSetArtifacts + resolveArtifactSHA |
hub/internal/web/configs.go (~L644 / ~L680) | POST /configuration/artifacts |
Operator UI to vouch artifact versions | With a Gitea client the sha is fetched AUTHORITATIVELY (submitted sha ignored); fetch failure refuses the save. Manual sha only in the no-creds fallback. |
(*gitea.Client).ListVersions / FileSHA256 |
hub/internal/gitea/gitea.go (~L47 / ~L72) | (ctx, pkg) ([]string, error) / (ctx, pkg, ver, file) |
Read-only Gitea generic-package metadata | sha comes from package metadata — artifact bytes are never downloaded. Newest-semver-first sort. |
(*Server).artifactChoices |
hub/internal/web/server.go (~L155) | (ctx, pkg, file) []artifactChoice |
Version+sha dropdown data | nil Gitea client / unreachable → nil → UI degrades to manual entry. One bad version drops itself, not the list. |
(*Handler).handleArtifactManifest |
hub/internal/api/handler.go (~L1550) | GET /api/v1/artifacts/{id} |
Serving the vouched set to the bootstrap script | Auth mirrors handleConfigRetrieve exactly. Unset manifest = empty fields, not an error. |
normalizeFloorInput / normalizeSHA256 |
hub/internal/web/configs.go (~L27 / ~L627) | (raw string) (string, bool) |
Validating operator-typed versions / shas | Empty string is VALID (means "clear"). Reuse for any new version/sha form field. |
PBS DR tier / tenantsync (hub/internal/tenantsync/, hub/internal/web/pbsdr.go, hub/internal/api/pbsdr.go)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
tenantsync.Client (Provision/Reissue/Fingerprint) |
hub/internal/tenantsync/client.go | (ctx, customerID) (*Result, error) |
ep0 per-customer PBS tenancy over the pinned-SSH forced-command channel (the wgsync twin) | Result.TokenSecret is transient custody → SaveHostPBSSecret immediately, never log the struct. Error paths NEVER embed stdout (the secret channel) — do not "improve" diagnostics by quoting the response. ErrTokenExists is typed: provision refuses an existing token; re-issue is the explicit path. |
(*Store).SaveHostPBSSecret / ConsumeHostPBSSecret |
hub/internal/store/pbsdr.go | (hostID, value) / (hostID) (string, error) |
HOST-scoped consume-once secret (the one_time_secrets host twin) | Same-tx mark-consumed; re-save resets consumption (re-issue supersedes). The agent consumes via POST /api/v1/hosts/{id}/pbs/consume-token (hub/internal/api/pbsdr.go). |
offsite.DeliveryStateFor (+ DeliveryStatus) |
hub/internal/offsite/delivery.go | (st, customerID) (DeliveryStatus, error) |
THE R-70 offsite last-mile detector — one implementation for every consumer (customer card deliveryViewFor, monitor.OffsiteDeliveryChecker event + R-71c heal) |
Precedence: applied (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + StaleStagedSince flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
(*Store).GetOneTimeSecretInfo / LastEventAt / LatestReportOffsitePresence / CountReportsOffsiteSince |
hub/internal/store/store.go | (customerID) (*OneTimeSecretInfo, error) / (customerID, eventType) (time.Time, error) / … |
Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | GetOneTimeSecretInfo never selects the value column — keep it that way. SetOneTimeSecretTimesForTest is the back-dating seam (PBSDR pattern). |
monitor.OffsiteDeliveryChecker + OffsiteReissuer |
hub/internal/monitor/offsite_delivery.go | NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger) |
R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in maybeHeal: re-reads the secret row at act time and refuses over an UNCONSUMED row — SaveOneTimeSecret clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
monitor.RestoreTestChecker + assessRestoreProven |
hub/internal/monitor/restoretest.go | NewRestoreTestChecker(st, onEvent, logger); .Check() |
R-85: turns a restore-test result into a SIGNAL — it was a [WARN] log line and nothing else, even for the tier already being tested |
TWO event types, never merged: restore_test_failed (broken now, error) vs restore_test_stale (unverified — not known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. Anchored on R-81 (assessRestoreProven reuses backupAssessment/verdict*): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: no customerMessages entry — do not add one without copy review. R-86 (2026-08-03): the window is PER TIER, not one constant. restoreProvenWindow(tier, observed, ok) = clamp(4 × max(observed, declared), floor 7d, cap 12d), where declared is that tier's own backup-freshness threshold (backupStaleAfter 26 h / offsiteBackupStaleAfter 8 d — reuse those, never a second opinion) and observed comes from observedArchiveIntervals over the retained window. Observation may only WIDEN: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. assessRestoreProven takes the window as an argument and every reason string names it (R-100's corollary). |
(*Server).applyPBSDR + mergePBSDR/readPBSDR |
hub/internal/web/pbsdr.go | (ctx, r, cfg) error |
The config form's DR-tier section → HOST desired_json pbs_dr descriptor + generation bump |
Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by cfg.DRTier (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
(*Server).pbsdrProvisionAtom + PBSDRAutoProvision |
hub/internal/web/pbsdr.go | (ctx, customerID, host, storageID) (blocked string, err error) / (ctx, customerID) |
The shared fresh-provision cascade atom; the WG-registration hook target (api SetWGRegisteredHook, wired in hub/cmd/hub/main.go when tenantsync is on) |
blocked != "" = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
cfg.DRTier + offsite coupling |
hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) | bool | Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) | One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field dr_tier (formBool helper). |
pbsdrheal.Reconciler + NewActions |
hub/internal/pbsdrheal/reconciler.go | NewReconciler(st, act, logger) · RestrictToHost(hostID) · Run(ctx)/Trigger() |
PBS-DR SELF-HEAL: re-arms a consumable secret for a box stuck in waiting_secret/consumed_failed after losing its converged marker (re-install/rollback). From SPIKE-pbsdr-selfheal-2026-07-15. |
Primary heal = re-stage the stored secret (no ep0 call, NO generation bump — a bump = agent refetch loop). Escalate to Re-issue only when no stored secret / consumed_failed. NEVER re-run pbsdrProvisionAtom (refuses ErrTokenExists) or blind-timer Re-issue (hash/gen thrash). Converged/disabled/verify_failed/DR-OFF = no-op. Debounce ≥2 distinct reports. PBSDRHEAL_ONLY_HOST scopes a supervised rollout. Fake seam: fakeActions in reconciler_test.go. |
(*Store).RestageHostPBSSecret + PBSDRHealStates |
hub/internal/store/pbsdr.go | (hostID) (restaged bool, err) / () ([]PBSDRHealRow, error) |
The self-heal store primitives: clear consumed_at IFF a row exists (re-arm the SAME value); the fleet work-set query (descriptor enable/provision + latest report pbs_dr.state + id) |
RestageHostPBSSecret does NO insert, NO value change, NO generation bump (restaged=false = no row → caller escalates). PBSDRHealStates mirrors GetHostOOBStates' latest-report-per-host join; malformed JSON → zero values, never an error. |
(*Server).ReissuePBSDR |
hub/internal/web/pbsdr.go | (ctx, customerID) error |
The non-HTTP core of the operator Re-issue button — the self-heal reconciler's escalation seam (satisfies pbsdrheal.Reissuer) |
Reuses tenantsync.Reissue + SaveHostPBSSecret + descriptor bump — NOT a re-run of pbsdrProvisionAtom. Keep in lockstep with the tail of handlePBSDRReissue (which is unchanged; the operator button's 303/400 codes are pinned by tests). |
parseHostCapabilities + capabilityView |
hub/internal/web/hosts.go | (reportJSON) []capabilityView |
Host-page capability chips (ok/degraded/inactive) | inactive (agent v0.86.0) = badge-neutral, NEVER warn/error — disabled ≠ degraded; unknown future statuses fall to the degraded styling (surface, don't hide). capabilitiesNeedDRMigration keys the pre-v1.15.0 migration hint on pbsdr-* + "binary not found". |
Config generation & secrets hygiene (hub/internal/configgen/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
configgen.Generate |
hub/internal/configgen/configgen.go (~L16) | (templateYAML, cfg) (string, error) |
Producing a customer controller.yaml | Programmatic overrides (customer id/hub url/api_key) ALWAYS win over config_json; fresh session secret per generation. |
configgen.RandomHex |
hub/internal/configgen/configgen.go (~L110) | (n int) (string, error) |
crypto/rand hex tokens (API keys, session secrets) | — |
configgen.RandomPassphrase |
hub/internal/configgen/passphrase.go (~L35) | (wordCount int) (string, error) |
Human-dictatable Hungarian passphrases (retrieval passwords) | ~29K-word embedded list; 5 words ≈ 74 bits. |
(*Store).EffectiveMinControllerVersion |
hub/internal/store/store.go (~L960) | (customerID) string |
Resolving the floor that actually applies (per-customer → global) | "" = no floor (Phase 2 inert). |
Assets, store misc, scheduling
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
assets.Manager (New, ServeFile, ReSeed) |
hub/internal/assets/assets.go (~L44/190/122) | seed-dir → PVC sync + manifest | Serving app logos/screenshots to controllers | ServeFile sanitizes to filepath.Base (no traversal). isAssetFile enforces the naming convention. |
copyFile |
hub/internal/assets/assets.go (~L259) | (src, dst) error |
THE atomic file write (tmp + rename) in the hub | Copy this shape for any new on-disk write. |
fileSHA256 |
hub/internal/assets/assets.go (~L244) | (path) (string, error) |
Streaming sha256 of a file | — |
(*Store).SaveEvent |
hub/internal/store/store.go (~L1003) | (...) (int64, error) |
Persisting ANY event (controller or hub source) | Pair with dispatcher/onEvent — saving alone never notifies. |
(*Store).RequestLogTail / GetPendingLogTailRequests / SaveAppLogTail |
hub/internal/store/logtail.go | pending-intent + consume-once fulfillment | THE ACK-flag pull pattern for hub→box requests (copy for any new one) | SaveAppLogTail clears the request in the SAME tx (consume-once) + prunes to last 2 per (customer,app); the hub NEVER connects into a box |
(*Store).RequestLogBundle / PendingLogBundleRequest / SaveLogBundle / PurgeExpiredLogBundles |
hub/internal/store/logbundle.go | component (controller/agent) log pulls — the v0.46.0 sibling of logtail.go | box-component debug-ring pulls; gzip custody, newest-3, 72 h TTL on the 60 s sweep | scope = customer_id (controller/report ACK) vs host_id (agent/heartbeat envelope); SaveLogBundle runs the SECRET GATE fail-closed (blocked flag row, no payload) and clears the request in the same tx; [REDACTED]/checksums pass by design |
upsertAppIssue dismissal/context semantics |
hub/internal/store/telemetry.go | ON CONFLICT CASE guards | Issue dismissal + first-capture-wins context | Un-dismiss ONLY on excluded.last_seen > dismissed_at; context adopted only while stored one is empty — do not "simplify" either CASE (red-proofed) |
store.GuestID |
hub/internal/store/store.go (~L1268) | (hostID string, vmid int) string |
Canonical guest primary key | Never hand-concatenate host+vmid. |
(*Store).GetHostReportsSince + GetFirstHostReportAt + monitor.newestBackupEvidence |
hub/internal/store/store.go, hub/internal/monitor/deadline.go | (customerID, since) ([]HostReportRow, error); (customerID) (time.Time, error); (rows, now) (time.Time, bool) |
Asking "when did the hub last SEE evidence of X?" instead of "what does the latest report say?" — the R-81 anchor. The agent's reporters are point-in-time and forget across a restart; the hub retains ~90 d of host-reports and does not. | The three go together: window scan + first-contact anchor + a bounded lookback (backupEvidenceLookback). Never judge a report-derived absence on the LATEST report alone — that is the bug class R-81 fixed for the third time. The scan early-exits on sufficiently-fresh evidence, so don't reorder rows away from newest-first. |
scheduleDaily |
hub/cmd/hub/main.go (~L449) | (ctx, name, "HH:MM", fn, logger) |
Daily jobs in Europe/Budapest (prune etc.) | Blocking — run as goroutine. parseHM returns 0,0 (midnight) on bad input. |
2. Canonical patterns (copy structure from THE named file)
| Pattern | Canonical file | Key traits |
|---|---|---|
| Monitor checker | hub/internal/monitor/staleness.go | Seed state on construction WITHOUT emitting events; in-memory states map under mutex; periodic Check(); emitTransition = SaveEvent then nil-checked onEvent; cleanup of vanished IDs. HostStaleness/HostDisk/HostLeaf/HostCapability/StorageFill all follow it. |
| API endpoint | hub/internal/api/handler.go ServeHTTP (~L139) + any handler |
Path switch in ServeHTTP; first line of every handler = checkAuth{Customer,Host}; io.LimitReader body cap; typed anonymous payload struct; explicit 4xx strings. |
| Web POST action | hub/internal/web/configs.go handleSetGlobalFloor (~L602) |
CSRF enforced centrally in web ServeHTTP; validate via normalize* helper; POST-redirect-GET with ?flash= token; log INFO on success. |
| Optional dependency injection | hub/internal/web/server.go Set* setters (~L131–148) / api handler SetDispatcher etc. |
Constructor takes hard deps; optional subsystems wired via SetX after construction; nil = graceful degradation (never panic). |
| Seam-injected sender for tests | hub/internal/notify/dispatcher.go sendEmailFn (~L33) |
Function-field defaulting to the real impl in the constructor; tests overwrite it. Same idea: mailRateLimiter.now, mailrelay.Sender fake. |
| Immediate-sync notify (per plane) | agent plane: poke.Notifier (web.Server.poke / api.Handler.poker via SetPoke/SetPoker) — controller plane: intent.Hub.Bump (web.Server.bumpIntent, api.Handler.intentHub) |
EVERY desired-state mutation fires the RIGHT plane's notifier AFTER the successful store write, never on an error path (fire-after-commit). Agent-plane pokes a HOST when its generation moved (SetHostDesired/Bump*HostDesired); controller-plane bumps a CUSTOMER on a controller-visible change. Both receivers COALESCE bursts — add NO hub-side dedup. Deliberate non-sites need a documented reason (undeliverable pre-tunnel, transport removed, or no generation bump → the 60 s ticker is the pickup path). Both seams nil-safe: unset = the ≤15-min cycle still reconciles. Full site inventory: REPORT.md audit table (v0.63.0). |
| Website page | website/index.html | UTF-8 with BOM; shared <nav>/<footer> byte-identical across pages (only class="active" differs); two-tone H1 = <h1>…<span>accent</span></h1>; all styling in website/assets/site.css tokens (:root) — zero embedded <style>; ?v=N cache-bust on site.css/icons.svg; umami snippet; no CDN fonts; no emoji (sprite icons.svg instead). |
| Gate script | scripts/site_gates.py | Byte-level mechanical gates (BOM, emoji codepoint ranges, nav/footer diff, analytics, banned tokens, cache-bust); run python scripts/site_gates.py after ANY website change; non-zero exit on failure. |
| Fetch-validate-install (shell) | scripts/felhom-host-install.sh step_agent_install (~L1108) |
fetch_raw to mktemp → syntax-check (bash -n) → install -m0755 -o root -g root → only then activate; guarded-mkfs wrapper installed BEFORE the sudoers that references it (ordering is the safety property). All mutations through run() (dry-run aware). |
| Install-profile gate (shell) | scripts/felhom-host-install.sh --mode appliance|byo (GL-2, v1.10.0) |
Mandatory-flag profile (no default), refusals at argv time BEFORE any prompt/step, risky step gated at its CALL SITE (one auditable place — never a branch inside the step), mode persisted to state.json + resume-mismatch refusal, FELHOM_INSTALL_STATE_DIR override for harness isolation. Harness: scripts/hostinstall-mode-harness.sh (static refusal matrix + grep-invariants + PVE dry-transcript tier; red-proofs run against a mutated scratch copy). |
| Disclosure↔uninstall parity (shell) | scripts/felhom-host-install.sh _uninstall_statement + harness GL4-D (v1.11.0) |
Every host artifact the byo disclosure names must be removed OR explicitly listed KEPT by run_uninstall; the harness greps the parity (token list). New install-time artifact ⇒ add its removal + disclosure line + parity token in the SAME commit. Drive data rule: plain umount only, never -l/-f, never any format op under /mnt/felhom-drives. |
| Website deploy (manifest) | manifests/webpage.yaml | git-sync sidecar (sparse-checkout /website/ + /scripts/, --link=current) + init container waits for first sync; nginx serves current/website; push to main = deployed, no image build. |
| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via secretKeyRef to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). report-api (the operator bearer, v0.53.0) is deliberately NOT optional: — a missing Secret fails Ready instead of booting an unauthenticatable hub. scripts/manifest_bearer_gate.py (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. ERRATA (2026-07-03): gitea-creds is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). |
| Hub deploy (GitOps) | manifests/hub.yaml image: (~L129) |
Pinned explicit tag, bumped in git, deliberate ArgoCD sync (auto-sync OFF). Code push alone deploys nothing. |
3. Dangerous lookalikes — do NOT reuse
| Trap | Why it bites | Use instead |
|---|---|---|
A plain missed bool for a report-derived absence (hub/internal/monitor/deadline.go) |
Collapsing the three-valued verdict re-introduces one of TWO failure modes: absence→MISSED is the 2026-07-26 cry-wolf (three boxes alarmed at once, one reached a customer channel); absence→OK means a genuinely dead box alarms NEVER, which is strictly worse. Three instances of this class so far: hub v0.12.0, v0.73.0, R-81. | `backupAssessment{verdict: verdictOK |
(*Handler).handleNotify + formatNotificationEmail + sendResendEmail (hub/internal/api/handler.go ~L1289/1624/1589) |
Legacy pre-dispatcher notification trio: no cooldowns, no operator channel, no allowedEventTypes gate, duplicate Hungarian formatter. Controller path is FROZEN until slice-10 cutover. | POST /api/v1/event → Dispatcher.ProcessEvent + notify.Format*Email |
Severity "critical" POSTed to a PRE-v0.31.0 hub |
Fixed in hub v0.31.0 (handleEvent now accepts critical). Older hubs coerce critical → "info", which never notifies — silent alert loss. Case-variants ("Critical") still coerce on every version. |
Against an old hub send warning/error; otherwise lowercase critical is safe |
compareVersions for anything security-ish (hub/internal/web/server.go ~L571) |
Returns 0 (equal) on unparseable input — a garbage version passes a floor check. gitea.compareSemver behaves differently (lexical fallback). |
Validate input with normalizeFloorInput first; then compareVersions is safe |
Inline stringData secrets à la manifests/felhom.secret.yaml |
Commits real credentials to git (healthchecks superuser pw, umami APP_SECRET/POSTGRES_PASSWORD, gitea-creds admin password still live there). | Out-of-band kubectl create secret + secretKeyRef (hub.yaml resend-api pattern; runbook documentation/runbooks/secrets.md) |
kubectl apply / kubectl set image on manifests/ |
ArgoCD app felhom reverts drift on next sync; live state lies about git. |
Edit manifest in git → push → ArgoCD sync (CLAUDE.md steps 3–5) |
:latest image tag in manifests |
Re-push doesn't change the manifest → no redeploy; Synced/Rollback misreport. | Pinned version tag, bumped per deploy |
| grep/regex hunting emoji in website HTML | Windows grep false-negatives multibyte emoji (proven in D0). | python scripts/site_gates.py (codepoint-range check) |
| Adding a website page without touching site_gates.py | PAGES list (scripts/site_gates.py ~L22) is explicit — an unlisted page is silently ungated (BOM/nav/emoji drift undetected). |
Add the filename to PAGES in the same commit |
4. Seams & interfaces (testing + cross-repo)
| Interface | Defined in | Implemented by | Fakes/tests at |
|---|---|---|---|
mailrelay.Sender |
hub/internal/mailrelay/relay.go (~L24) | ResendSMTP (prod) |
fake sender in hub/internal/api/mail_test.go; hub/internal/mailrelay/relay_test.go |
Dispatcher.sendEmailFn (func seam) |
hub/internal/notify/dispatcher.go (~L33) | (*Dispatcher).sendEmail (Resend HTTP) |
hub/internal/notify/dispatcher_test.go |
monitor.EventNotifyFunc |
hub/internal/monitor/staleness.go (~L14) | closure over Dispatcher.ProcessEvent (main.go) |
hub/internal/monitor tests (captured-events func) |
api.ConfigTemplateProvider |
hub/internal/api/handler.go (~L24) | web.TemplateFetcher (Gitea-pulled controller.yaml template) |
stub providers in api tests |
api.LatestVersionProvider |
hub/internal/api/handler.go (~L31) | web.VersionChecker (registry poll) |
hub/internal/api/config_version_ack_test.go |
mailRateLimiter.now (func seam) |
hub/internal/api/mail.go (~L27) | time.Now |
hub/internal/api/mail_test.go clock injection |
web.tenancyProvisioner |
hub/internal/web/pbsdr.go | *tenantsync.Client (pinned SSH to ep0's felhom-tenantsync) |
fakeTenancy in hub/internal/web/pbsdr_test.go; in-process SSH server in hub/internal/tenantsync/client_test.go |
| Cross-repo: ep0 tenancy surface | scripts/felhom-tenantsync.sh (JSON stdin/stdout forced command) |
installed on ep0 per runbook offsite-endpoint.md §10 | provision/reissue/fingerprint ops; token secret rides stdout ONLY; the peersync script/key are untouched |
| Cross-repo: controller → hub | POST /api/v1/report (frozen) + POST /api/v1/event |
felhom-controller repo | new event types MUST enter allowedEventTypes (hub/internal/api/handler.go ~L1063) or the controller gets 400 |
| Cross-repo: agent → hub | POST /api/v1/host-report, /host-enroll, jobs/desired-state/escrow routes (handler.go ~L145–189) |
felhom-agent repo | hub/internal/api/host_test.go, desired_test.go, escrow_test.go, dr_test.go |
| Cross-repo: Day-0 bootstrap → hub | GET /api/v1/config/{id} + /artifacts/{id} (X-Retrieval-Password) |
scripts/felhom-host-install.sh (fetches + sha256-verifies against the hub-vouched manifest) | hub/internal/api/artifact_test.go |
| Cross-repo: controller ← hub assets | GET /api/v1/assets/manifest + /assets/file/{name} |
felhom-controller pulls app logos/screenshots | assets manifest sha-based change detection |
5. Extension points (where new features plug in)
- New event type — THREE registers, and which ones depend on the AUDIENCE. Always:
allowedEventTypes(hub/internal/api/handler.go) — missing it means the controller's POST 400s and the event vanishes (the known gotcha). Then decide the audience and stop guessing from the other registers:- Operator-only → add to
notify.operatorOnlyEvents(hub/internal/notify/dispatcher.go) and give it nocustomerMessagesentry. Allowlisting alone does NOT make a type operator-only —FormatCustomerEmailtreats a missingcustomerMessagesentry as a fallback to the raw message, not a block, and the only customer gate is configuration. v0.78.0 asserted the opposite in a comment and shipped the defect (R-97c). Examples:whole_guest_backup_failed,recovery_unit_capture_failed. - Customer-facing with a STATIC message → add a
customerMessagesentry (hub/internal/notify/templates.go) and the controller'ssettings.DefaultEnabledEventsif it should be on by default. - Customer-facing with a DYNAMIC message (the producer builds Hungarian text carrying names/numbers) → deliberately no
customerMessagesentry:FormatCustomerEmailPREFERS the entry over the message, so adding one silently discards the specifics. Examples:offbox_enlarge_blocked,disk_health_degraded, and since v0.89.0disk_warning/disk_critical. - Pin BOTH registers in ONE test (hub/internal/api/recovery_unit_event_test.go is the model) — fixing one and not the other is the realistic mistake, and
notify.IsOperatorOnlyexists so the api package can assert it. - A type in these registers with no PRODUCER is inert.
disk_warning/disk_criticalwere allowlisted, copy'd, default-enabled and checkbox'd from early on, and nothing in any repo emitted them until controller v0.191.0 — grep for an emitter before assuming a type works.
- Operator-only → add to
- New monitor checker: copy hub/internal/monitor/staleness.go (§2 pattern); wire in hub/cmd/hub/main.go with an
EventNotifyFunc; severity must be warning/error/critical to notify. - New API route: switch in
api.ServeHTTP(handler.go ~L139); auth helper first line. - New web page/action: switch in
web.ServeHTTP(server.go ~L182) — non-GET gets CSRF automatically; template into hub/internal/web/templates/ (embedded FS, parsed inweb.New); new helpers into the funcMap (server.go ~L67). - New template func: funcMap in web.New only; add a case to hub/internal/web/funcmap_test.go.
- New daily job:
scheduleDailyin hub/cmd/hub/main.go + add pruning topruneAllif data grows. - New site gate: append to scripts/site_gates.py; new website pages go into its
PAGESlist. - New artifact kind (Day-0): consts
pkg*/file*(hub/internal/web/server.go ~L27),ArtifactManifestfields + settings keys (hub/internal/store/store.go ~L905),handleSetArtifacts,artifactManifestResponse(handler.go), and the install script's verify step. - New host-install step:
step_*function in scripts/felhom-host-install.sh usingrun()/fetch_raw/diehelpers; keep dry-run coverage. - New DR-recipe section:
hostHalfShape/appHalfShapeandAssembledRecipe(hub/internal/store/dr_recipe.go) — those shape structs are ALLOW-LISTS, not forward-compat: a section only the emitter knows about is stored intact and silently dropped before the operator downloads it. No error, no log, no red test. That is R-122: the controller emittedoffsite_resticfrom fork-4, all three real customers had it stored, and no delivered recipe ever contained it. Then extendTestAssembleDRRecipe_CarriesEveryEmittedSection(same commit) and, for a host-half section, the agent'sDRRecipeHostHalf+ BOTH copies oftestdata/host-report.golden.json(byte-identical, cross-repo).
6. Known duplication (observed — NOT fixed)
- Resend HTTP sender ×2:
(*Handler).sendResendEmail(hub/internal/api/handler.go ~L1589) ≈(*Dispatcher).sendEmail(hub/internal/notify/dispatcher.go ~L185) — byte-near-identical POST to api.resend.com. Kept because the handler copy serves the frozen legacy /notify path. - Hungarian customer-email formatter ×2:
formatNotificationEmail(hub/internal/api/handler.go ~L1624) vsnotify.FormatCustomerEmail(hub/internal/notify/templates.go ~L118). Legacy vs dispatcher; the legacy one lacks the per-event-type message map. - Semver compare ×2 with DIFFERENT fallback semantics:
web.compareVersions(hub/internal/web/server.go ~L571, parse error → 0) vsgitea.compareSemver(hub/internal/gitea/gitea.go ~L115, parse error → lexical). Documented as deliberate (import-cycle avoidance) in gitea.go, but the behavior drift is not. - Checker-family structural repetition: staleness.go vs host_staleness.go, and host_disk.go vs storage_fill.go (band/bandRank/bandLabel vs bandForPercent) — same skeleton re-implemented per domain; treated as the accepted §2 pattern rather than a defect.
- Duration formatting ×2:
monitor.formatDuration(hub/internal/monitor/staleness.go ~L187) vsweb.timeAgo(hub/internal/web/server.go ~L603) — different audiences (email vs UI) but overlapping logic.