Files
felhom.eu/REUSE.md
T
admin e07d90f0f4 hub v0.85.0 — Network card: a host's addresses are visible at last
Pairs with agent v0.119.0 and is useless without it.

A managed box's LAN IP was not shown anywhere in the hub, because nothing
reported it — the host report carried no address of any kind. The only IP
reachable from the UI at all was the WireGuard one, on /offsite's peer table
keyed by pubkey, so an operator could go peer->host and never host->peer, which
is the direction anyone actually asks in.

The host page grows a Network card: every routable address the box holds, one row
per (interface, address), plus a WireGuard row. On demo-felhom that is vmbr0
192.168.0.162/24 and tailscale0 100.70.170.35/32 — with the PVE web console at
https://<the LAN address>:8006, the thing the operator wanted and could not get.

WireGuard is rendered as TWO facts, deliberately. WGAssignedIP is the hub's own
allocation (wg_peers, authoritative desired state); WGConfirmed is whether the box
reports actually holding it. Showing the allocation alone would make a peer that
was never applied look healthy — the same shape as reading a timestamp that
records an attempt as if it recorded a result.

The split is keyed on the ALLOCATION, not the interface name: wg-felhom is the
agent's current unit name, and a UI keyed on that string would silently
mis-render the day it changes.

An old agent renders UNKNOWN, never "no addresses". Below agent 0.119.0 the field
is absent from the wire, and an absent signal is not a negative result — the page
says so and names the version needed. Rendering an empty list there would have
stated something false about the host.

No new store table and no new ingest path: the report is already stored opaquely
and GetWGPeerForHost already existed with no UI consumer. This is parse + render.

The report fixture in the tests is the REAL wire — the addresses block copied out
of `felhom-agent --selftest=hub` on demo-felhom running 0.119.0.

Tests 559 -> 566; four red-proofs (inert view-model, unconditional confirmation,
the old-agent branch, and the drift case) each run, observed failing, reverted.
2026-07-31 08:49:44 +02:00

46 KiB
Raw Blame History

REUSE.md — felhom.eu (hub + website + scripts)

Before writing new code, check here. Canonical helpers, patterns to copy, traps to avoid. Maintenance: update in the SAME commit that adds/changes/deprecates a shared helper. Entries cite file + symbol. Line numbers are landmarks only — reconfirm before editing.

1. Canonical helpers (MUST reuse — do not reinvent)

Report ingest & API auth (hub/internal/api/)

Symbol File Short signature Use for Gotchas
(*Handler).checkAuthCustomer hub/internal/api/handler.go (~L94) (r) (customerID string, isGlobal, ok bool) Bearer auth for controller-facing endpoints (global key OR per-customer key) Global key → ("", true, true): caller must then trust body customer_id. Constant-time compare on global key.
(*Handler).checkAuthHost hub/internal/api/handler.go (~L119) (r) (hostID, customerID string, isGlobal, ok bool) Bearer auth for agent-facing endpoints (global OR per-host key) Sibling of checkAuthCustomer — do NOT mix the two token namespaces. Global key requires the host row to already exist (see handleHostReport).
(*Handler).handleEvent + allowedEventTypes hub/internal/api/handler.go (~L1115 / ~L1063) POST /api/v1/event The ONLY controller→hub structured-event ingest Unknown event_type → 400 (add to the map FIRST). Accepted severities: info/warning/error/critical (critical since v0.31.0); anything else coerces to "info" — exact-match lowercase ("Critical" coerces). Tests: hub/internal/api/event_test.go.
(*Handler).handleHostReport hub/internal/api/handler.go (~L464) POST /api/v1/host-report Agent heartbeat ingest: denorm + guest upsert Body cap via LimitReader; per-host key enforces host_id match (403 on mismatch); received_at is the dead-man's-switch.
(*Handler).handleConfigRetrieve hub/internal/api/handler.go (~L1484) GET /api/v1/config/{id}, header X-Retrieval-Password Canonical password-gated retrieval endpoint Constant-time compare vs cfg.RetrievalPassword; 404-before-401 ordering. handleArtifactManifest mirrors it EXACTLY — keep them in lockstep. Also the Day-0 claim entry point: calls claimEngine.EnsureIssued + bakes the hash via configgen.Generate(…, claimState).
claim.Engine hub/internal/claim/engine.go EnsureIssued / Resend / RequestReset / MarkClaimed (all take *store.CustomerConfig) Customer-claim code engine (v0.50.0, F-4) Stores bcrypt(code) ONLY — plaintext lives just in the email send. EnsureIssued is idempotent (never rotates/re-sends an existing row). Wired via api.SetClaimEngine + web.SetClaimEngine; the Mailer seam is *notify.Dispatcher.
(*Store).RotateClaimCode / GetClaim / MarkClaimed hub/internal/store/store.go claim-state CRUD customer_claims row (v0.50.0) RotateClaimCode bumps generation (single active code) + PRESERVES claimed_at (reset never un-claims); MarkClaimed is set-only.
configgen.Generate hub/internal/configgen/configgen.go (~L16) (templateYAML string, cfg *store.CustomerConfig, claimState *store.ClaimState) (string, error) Generate a customer's controller.yaml The 3rd arg (nil-safe) bakes web.claim_code_*. The REAL config-retrieve path issues+emails first (EnsureIssued); the preview/DR paths bake read-only via store.GetClaim.
notify.FormatClaimEmail / (*Dispatcher).SendClaimEmail hub/internal/notify/{templates,dispatcher}.go (kind, customerID, email, domain, code) Hungarian claim/reset/claimed emails kind ∈ claim|reset|claimed. The dispatcher method IS the claim.Mailer. Never log the code.
writeJSON hub/internal/api/dr.go (~L25) (w, code int, v any) JSON responses in api package Only used in dr.go so far; prefer it over ad-hoc byte-writes for new endpoints.
(*mailRateLimiter).allow hub/internal/api/mail.go (~L48) (key string) bool Per-key token-bucket rate limiting Refill = perMinute/60 per sec, burst = perMinute; in-memory (lost on restart, accepted). now is injectable for tests.

Alerting / Resend / dispatcher (hub/internal/notify/, monitor/)

Symbol File Short signature Use for Gotchas
(*Dispatcher).ProcessEvent hub/internal/notify/dispatcher.go (~L88) (customerID, eventType, severity, message, detailsJSON, source) THE notification pipeline (operator + customer channels, cooldowns, prefs) Call in a goroutine (handlers do go d.ProcessEvent(...)). No Resend key → silent no-op. eventType=="test" bypasses prefs/cooldown (and since v0.71.0 also mails the operator). *_recovered routes via the explicit recovery branch BEFORE the severity gate (v0.71.0) — see processRecovery.
(*Dispatcher).processRecovery hub/internal/notify/dispatcher.go (~L160) internal node_recovered/host_recovered routing (audit F11) Operator always (1 h cooldown); customer iff PAIRED — a customer-channel sent stale/down row newer than the last sent recovery (store.LastCustomerSentAt); enabled_events deliberately ignored for recovery; timestamp ties → no mail (flap-safe). Severity stays info — never "fix" that by widening severityNotifies.
severityNotifies hub/internal/notify/dispatcher.go (~L77) (severity string) bool Deciding whether a severity emails warning/error/critical notify; info intentionally doesn't; anything else is logged as unrecognized (v0.24.0 fix — do not regress). Recovery mails exist DESPITE this gate (eventType branch), not through it.
priorityHeaders hub/internal/notify/dispatcher.go (~L56) (severity string) map[string]string High-priority mail-client nudge (audit F14-light) error/critical → X-Priority: 1 + Importance: high; everything else nil — a warning/info mail must NOT masquerade as urgent (red-proofed).
sendEmailFn seam / sendEmail hub/internal/notify/dispatcher.go (~L33 / ~L300) func(to, subject, textBody string, headers map[string]string) error Test seam for all sends; Resend POST Signature grew a headers param in v0.71.0 — payload carries "headers" only when non-empty. Tests capture recipient+subject+headers through the seam.
FormatOperatorEmail / FormatCustomerEmail hub/internal/notify/templates.go (~L24 / ~L118) (...) (subject, body) Operator (English) / customer (Hungarian) email bodies Customer messages come from the customerMessages map — add the Hungarian text when adding an event type. Budapest TZ via package init(). Operator icon is eventType-aware: *_recovered (severity is the fallback).
monitor.EventNotifyFunc hub/internal/monitor/staleness.go (~L14) func(customerID, eventType, severity, message, detailsJSON, source) Decoupling checkers from notify; wired to dispatcher.ProcessEvent in main May be nil — always nil-check before calling (all checkers do).
(*Store).LogNotification hub/internal/store/store.go (~L433) (customerID, eventType, severity, message, status, errorMsg, channel) Audit trail of every send attempt (sent/failed, per channel) Log BOTH success and failure (dispatcher does). Since v0.71.0 these rows are also the recovery PAIRING evidence — never prune them casually.
(*Store).LastCustomerSentAt hub/internal/store/store.go (~L815) (customerID, eventTypes []string) (time.Time, bool, error) Pairing-evidence query (max customer-channel sent created_at over types) Uses the (customer_id, created_at DESC) index. Empty type list → (zero, false, nil).
(*Store).SeedNotificationPrefs hub/internal/store/store.go (~L850) (customerID, email, enabledEvents) (seeded bool, err) Claim-time prefs seeding (audit F12) INSERT OR IGNORE — never an upsert (red-proofed); empty email = no-op. Customer edits go through SaveNotificationPrefs, seeds NEVER do.

App-mail passthrough (hub/internal/mailrelay/)

Symbol File Short signature Use for Gotchas
mailrelay.Sender / (*ResendSMTP).Send hub/internal/mailrelay/relay.go (~L24 / ~L50) Send(ctx, raw []byte, mailFrom, rcptTo) error Raw-MIME passthrough to Resend SMTP Deliberately separate from notify's HTTP path — parse-then-API drops inline CID images (spike-proven). Do NOT "unify" them. Delivery verdict lands at DATA-close.
mailrelay.FromDomain hub/internal/mailrelay/relay.go (~L130) (raw []byte) (string, error) From-HEADER domain extraction for allowlisting Header domain, not envelope — Resend checks the header.

Web auth / session / CSRF (hub/internal/web/)

Symbol File Short signature Use for Gotchas
(*Server).RequireAuth hub/internal/web/server.go (~L359) (next http.Handler) http.Handler Session-cookie OR Basic-auth gate for all web routes Empty effective hash disables auth entirely (dev mode). Browsers → /login redirect; JSON-ish requests → 401.
(*Server).effectivePasswordHash hub/internal/web/server.go (~L118) () string THE single source for the operator login hash — call this, never read configPasswordHash Precedence: hub_settings DB override (set via Configuration UI) wins, else the hub.yaml auth.password_hash seed. ConfigMap = break-glass reset. Change it via POST /configuration/password (handleChangePassword).
(*Server).validateCSRF hub/internal/web/server.go (~L446) (r) bool CSRF check — enforced centrally in web.ServeHTTP for every non-GET No session cookie → returns true (Basic-auth path is exempt). New POST routes get CSRF for free; forms MUST embed csrfField.
(*Server).csrfField hub/internal/web/server.go (~L483) (r) template.HTML Hidden _csrf input for HTML forms Pass into template data on every form-rendering handler.
(*Server).CleanupSessions hub/internal/web/server.go (~L110) (ctx) — goroutine Expired-session sweeper Started once from main; 15-min tick.

Status tokens & template funcmap (hub/internal/web/)

Symbol File Short signature Use for Gotchas
funcMap in web.New hub/internal/web/server.go (~L67) template.FuncMap{...} ALL template helpers (timeAgo, statusColor, json, hubVersion…) Add new template funcs HERE, nowhere else. Tested by hub/internal/web/funcmap_test.go + render_test.go.
inline_confirm_js partial (v0.52.0) hub/internal/web/templates/inline_confirm.html {{template "inline_confirm_js"}} in the page <head> Inline "question + Igen/Mégse" confirm for consequential buttons — data-confirm="…" on the button, or felhomConfirm(el, q, onYes) from JS NEVER native confirm()/prompt() — OS-modals freeze browser automation (F-16). scripts/hub_confirm_gate.py asserts zero, but is not wired to run (R-29) — so the rule holds only as long as you keep it. Uses requestSubmit so formaction sub-buttons riding a parent form work. NOT for the danger-zone typed-confirm cascade.
timeAgo hub/internal/web/server.go (~L603) (t time.Time) string Human-relative timestamps in UI
statusColor hub/internal/web/server.go (~L630) (status string) string Status → design-system-v2 token (nominal/warn/crit/neutral) Class SUFFIX only, never inline color (D4). Exception-color principle: healthy = blue/neutral.
(*Server).hostStatus + hostStatusClass/hostStatusLabel hub/internal/web/hosts.go (~L16/34/48) (lastReport *time.Time) string Host liveness badge Uses the SAME threshold as HostStalenessChecker (down = 2× stale) — never invent a second definition.
parseSQLiteTime hub/internal/store/store.go (~L1160) (s string) time.Time Parsing ANY timestamp read from SQLite modernc/sqlite returns multiple formats; raw time.Parse will intermittently zero out. Always use this.
compareVersions hub/internal/web/server.go (~L571) (a, b string) int X.Y.Z comparisons in web (floor checks, update-available) Returns 0 on parse error — unparseable compares as "equal" (see §3).

Host views & lifecycle / offsite endpoints (v0.47.0, hub/internal/web + store)

Symbol File Short signature Use for Gotchas
(*Server).hostDetailData hub/internal/web/hosts.go (~L282) (host *store.Host, r) map[string]interface{} The ONE view-model builder for the shared host_detail_body sub-template (standalone /hosts/{id} + customer Host tab) Booleans/counts only for DR/escrow; carries Deletable (= status != "ok") which gates the danger-zone card. Never add a secret field.
parseHostAddresses + (*Server).hostNetwork / hostNetworkView (v0.85.0) hub/internal/web/hosts.go (reportJSON) []hostAddressView · (host, reportJSON) hostNetworkView The host page's Network card: every routable address the box holds + its WireGuard allocation Needs agent >= 0.119.0 (minAgentForAddresses); below it the wire has no addresses key and the card renders UNKNOWN, never "no addresses" — an absent signal is not a negative result. WireGuard is TWO facts: the hub's allocation (GetWGPeerForHost, authoritative) AND whether the box confirms holding it — the allocation alone cannot distinguish a live tunnel from a peer that was never applied. The WG row is split out by comparing against the ALLOCATION, never by matching the interface name wg-felhom, which is a unit name that can change.
(*Store).GetHostRecoveryMeta + (*Server).handleHostRevealRecoveryCredential hub/internal/store/host_recovery.go · hub/internal/web/hosts.go (hostID) (*HostRecoveryMeta, error) · POST /hosts/{id}/reveal-recovery-credential The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) Use GetHostRecoveryMeta on any page-render path — its struct and its SELECT both omit the secret column, so it cannot leak one; GetHostRecoveryCredential (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE recovery_credential_revealed event via SaveEvent and calls NO dispatcher (the handleRequestLogTail shape). api/handler.go handleAdminGetRecoveryCredential (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133.
host_detail_body sub-template hub/internal/web/templates/host_detail_body.html {{template "host_detail_body" .}} Rendering a host's detail sections on ANY surface One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with {{.HostID}} (the customer page renders N instances).
(*Store).ListHostsByCustomer hub/internal/store/store.go (~L1620) (customerID) ([]Host, error) A customer's hosts, host_id order A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer.
(*Server).configFormData (v0.49.0) hub/internal/web/configs.go (~L430) (r, isNew, cfg, overrides, errMsg) configFormView The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) overrides=nil → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed).
config_form_body sub-template (v0.49.0) hub/internal/web/templates/config_form_body.html {{template "config_form_body" <configFormView>}} Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) The floor/geo/danger cards on the Edit tab are SIBLINGS after </form> — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight <script>.
(*Store).CountHostArtifacts / DeleteHost hub/internal/store/store.go (~L1640/~L1690) (hostID) (HostArtifacts, error) / (hostID, deleteEscrow bool) error Host-delete impact preview + the ONE-transaction cascade ONLINE gate lives in the handler, escrow gate in the store (ErrHostEscrowPresent, tx never starts). log_bundles die by scope_id == host_id ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out.
(*Server).commitCustomerReset (v0.69.0) hub/internal/web/customer_reset.go (~L165) (ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the customer_resets journal Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). purgeEscrow governs ONLY whether PurgeCustomerResetDBState destroys retained custody: standalone RESET passes the operator's escrow_ack; the DELETE cascade passes false so custody dies exactly once, in its leg 3. Returns a resetLegError carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them.
(*Store).CustomerResidue / PurgeCustomerResidue (v0.70.0) hub/internal/store/customer_delete.go (customerID) (*CustomerResidue, error) / (customerID) error Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind GetCustomers() is REPORT-derived — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared residueQueries list so a table can never be counted-but-not-purged. Includes appliance_registrations + selfbind_tokens (credential-bearing, not telemetry). NEVER touches events, notification_log, host_deletions, customer_resets.
(*Server).handleCustomerDelete / handleCustomerDeletePreview (v0.69.0) hub/internal/web/customer_delete.go (w, r, customerID) THE customer offboarding entry: the guided full-teardown cascade hosts → RESET → purge (R-25b). GET = live inventory JSON for the dialog, POST = the cascade There is NO shallow delete path any more — the old handleConfigDelete is gone; do not reintroduce one. Every gate (3 acks, typed customer-id, stale host-count, ONLINE-host refusal) runs BEFORE any write, so a refusal has zero side effects. Leg order is load-bearing twice over: ruling 3 (RESET never sees a host row) and custody purged exactly once, in leg 3. A failed leg retains the journal — a re-run resumes and must pass every gate again.
(*Store).ListWGEndpoints / DeleteWGEndpoint hub/internal/store/wg.go (~L64/~L86) () ([]WGEndpoint, error) / (endpointID) error The /offsite endpoint-management surface GetWGEndpoint (lowest id, LIMIT 1) stays THE allocation/sync endpoint — do not switch allocator/reconciler/desired-state to the list without the wg_peers.endpoint_id migration arc. Peers-in-subnet guards live in hub/internal/web/offsite.go.

Artifact manifest / Day-0 trust root

Symbol File Short signature Use for Gotchas
(*Store).GetArtifactManifest / SetArtifactManifest hub/internal/store/store.go (~L933 / ~L944) () ArtifactManifest / (m) error The DB-backed (hub_settings) Day-0 artifact record This is the checksum TRUST ROOT the host-bootstrap verifies against — distinct from Gitea, which only stores bytes.
(*Store).GetOperatorPasswordHash / SetOperatorPasswordHash hub/internal/store/store.go (~L1350) () string / (hash) error The DB-backed (hub_settings) operator login password override Read via Server.effectivePasswordHash(), not directly. "" = no override (config seed authoritative). Store the bcrypt hash, never the plaintext.
(*Server).handleSetArtifacts + resolveArtifactSHA hub/internal/web/configs.go (~L644 / ~L680) POST /configuration/artifacts Operator UI to vouch artifact versions With a Gitea client the sha is fetched AUTHORITATIVELY (submitted sha ignored); fetch failure refuses the save. Manual sha only in the no-creds fallback.
(*gitea.Client).ListVersions / FileSHA256 hub/internal/gitea/gitea.go (~L47 / ~L72) (ctx, pkg) ([]string, error) / (ctx, pkg, ver, file) Read-only Gitea generic-package metadata sha comes from package metadata — artifact bytes are never downloaded. Newest-semver-first sort.
(*Server).artifactChoices hub/internal/web/server.go (~L155) (ctx, pkg, file) []artifactChoice Version+sha dropdown data nil Gitea client / unreachable → nil → UI degrades to manual entry. One bad version drops itself, not the list.
(*Handler).handleArtifactManifest hub/internal/api/handler.go (~L1550) GET /api/v1/artifacts/{id} Serving the vouched set to the bootstrap script Auth mirrors handleConfigRetrieve exactly. Unset manifest = empty fields, not an error.
normalizeFloorInput / normalizeSHA256 hub/internal/web/configs.go (~L27 / ~L627) (raw string) (string, bool) Validating operator-typed versions / shas Empty string is VALID (means "clear"). Reuse for any new version/sha form field.

PBS DR tier / tenantsync (hub/internal/tenantsync/, hub/internal/web/pbsdr.go, hub/internal/api/pbsdr.go)

Symbol File Short signature Use for Gotchas
tenantsync.Client (Provision/Reissue/Fingerprint) hub/internal/tenantsync/client.go (ctx, customerID) (*Result, error) ep0 per-customer PBS tenancy over the pinned-SSH forced-command channel (the wgsync twin) Result.TokenSecret is transient custody → SaveHostPBSSecret immediately, never log the struct. Error paths NEVER embed stdout (the secret channel) — do not "improve" diagnostics by quoting the response. ErrTokenExists is typed: provision refuses an existing token; re-issue is the explicit path.
(*Store).SaveHostPBSSecret / ConsumeHostPBSSecret hub/internal/store/pbsdr.go (hostID, value) / (hostID) (string, error) HOST-scoped consume-once secret (the one_time_secrets host twin) Same-tx mark-consumed; re-save resets consumption (re-issue supersedes). The agent consumes via POST /api/v1/hosts/{id}/pbs/consume-token (hub/internal/api/pbsdr.go).
offsite.DeliveryStateFor (+ DeliveryStatus) hub/internal/offsite/delivery.go (st, customerID) (DeliveryStatus, error) THE R-70 offsite last-mile detector — one implementation for every consumer (customer card deliveryViewFor, monitor.OffsiteDeliveryChecker event + R-71c heal) Precedence: applied (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + StaleStagedSince flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS.
(*Store).GetOneTimeSecretInfo / LastEventAt / LatestReportOffsitePresence / CountReportsOffsiteSince hub/internal/store/store.go (customerID) (*OneTimeSecretInfo, error) / (customerID, eventType) (time.Time, error) / … Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) GetOneTimeSecretInfo never selects the value column — keep it that way. SetOneTimeSecretTimesForTest is the back-dating seam (PBSDR pattern).
monitor.OffsiteDeliveryChecker + OffsiteReissuer hub/internal/monitor/offsite_delivery.go NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger) R-70 stuck event + R-71c self-heal on the shared 60 s ticker THE R-39(a) GUARD lives in maybeHeal: re-reads the secret row at act time and refuses over an UNCONSUMED row — SaveOneTimeSecret clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op.
monitor.RestoreTestChecker + assessRestoreProven hub/internal/monitor/restoretest.go NewRestoreTestChecker(st, onEvent, logger); .Check() R-85: turns a restore-test result into a SIGNAL — it was a [WARN] log line and nothing else, even for the tier already being tested TWO event types, never merged: restore_test_failed (broken now, error) vs restore_test_stale (unverified — not known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. Anchored on R-81 (assessRestoreProven reuses backupAssessment/verdict*): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: no customerMessages entry — do not add one without copy review.
(*Server).applyPBSDR + mergePBSDR/readPBSDR hub/internal/web/pbsdr.go (ctx, r, cfg) error The config form's DR-tier section → HOST desired_json pbs_dr descriptor + generation bump Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by cfg.DRTier (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy.
(*Server).pbsdrProvisionAtom + PBSDRAutoProvision hub/internal/web/pbsdr.go (ctx, customerID, host, storageID) (blocked string, err error) / (ctx, customerID) The shared fresh-provision cascade atom; the WG-registration hook target (api SetWGRegisteredHook, wired in hub/cmd/hub/main.go when tenantsync is on) blocked != "" = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration.
cfg.DRTier + offsite coupling hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) bool Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field dr_tier (formBool helper).
pbsdrheal.Reconciler + NewActions hub/internal/pbsdrheal/reconciler.go NewReconciler(st, act, logger) · RestrictToHost(hostID) · Run(ctx)/Trigger() PBS-DR SELF-HEAL: re-arms a consumable secret for a box stuck in waiting_secret/consumed_failed after losing its converged marker (re-install/rollback). From SPIKE-pbsdr-selfheal-2026-07-15. Primary heal = re-stage the stored secret (no ep0 call, NO generation bump — a bump = agent refetch loop). Escalate to Re-issue only when no stored secret / consumed_failed. NEVER re-run pbsdrProvisionAtom (refuses ErrTokenExists) or blind-timer Re-issue (hash/gen thrash). Converged/disabled/verify_failed/DR-OFF = no-op. Debounce ≥2 distinct reports. PBSDRHEAL_ONLY_HOST scopes a supervised rollout. Fake seam: fakeActions in reconciler_test.go.
(*Store).RestageHostPBSSecret + PBSDRHealStates hub/internal/store/pbsdr.go (hostID) (restaged bool, err) / () ([]PBSDRHealRow, error) The self-heal store primitives: clear consumed_at IFF a row exists (re-arm the SAME value); the fleet work-set query (descriptor enable/provision + latest report pbs_dr.state + id) RestageHostPBSSecret does NO insert, NO value change, NO generation bump (restaged=false = no row → caller escalates). PBSDRHealStates mirrors GetHostOOBStates' latest-report-per-host join; malformed JSON → zero values, never an error.
(*Server).ReissuePBSDR hub/internal/web/pbsdr.go (ctx, customerID) error The non-HTTP core of the operator Re-issue button — the self-heal reconciler's escalation seam (satisfies pbsdrheal.Reissuer) Reuses tenantsync.Reissue + SaveHostPBSSecret + descriptor bump — NOT a re-run of pbsdrProvisionAtom. Keep in lockstep with the tail of handlePBSDRReissue (which is unchanged; the operator button's 303/400 codes are pinned by tests).
parseHostCapabilities + capabilityView hub/internal/web/hosts.go (reportJSON) []capabilityView Host-page capability chips (ok/degraded/inactive) inactive (agent v0.86.0) = badge-neutral, NEVER warn/error — disabled ≠ degraded; unknown future statuses fall to the degraded styling (surface, don't hide). capabilitiesNeedDRMigration keys the pre-v1.15.0 migration hint on pbsdr-* + "binary not found".

Config generation & secrets hygiene (hub/internal/configgen/)

Symbol File Short signature Use for Gotchas
configgen.Generate hub/internal/configgen/configgen.go (~L16) (templateYAML, cfg) (string, error) Producing a customer controller.yaml Programmatic overrides (customer id/hub url/api_key) ALWAYS win over config_json; fresh session secret per generation.
configgen.RandomHex hub/internal/configgen/configgen.go (~L110) (n int) (string, error) crypto/rand hex tokens (API keys, session secrets)
configgen.RandomPassphrase hub/internal/configgen/passphrase.go (~L35) (wordCount int) (string, error) Human-dictatable Hungarian passphrases (retrieval passwords) ~29K-word embedded list; 5 words ≈ 74 bits.
(*Store).EffectiveMinControllerVersion hub/internal/store/store.go (~L960) (customerID) string Resolving the floor that actually applies (per-customer → global) "" = no floor (Phase 2 inert).

Assets, store misc, scheduling

Symbol File Short signature Use for Gotchas
assets.Manager (New, ServeFile, ReSeed) hub/internal/assets/assets.go (~L44/190/122) seed-dir → PVC sync + manifest Serving app logos/screenshots to controllers ServeFile sanitizes to filepath.Base (no traversal). isAssetFile enforces the naming convention.
copyFile hub/internal/assets/assets.go (~L259) (src, dst) error THE atomic file write (tmp + rename) in the hub Copy this shape for any new on-disk write.
fileSHA256 hub/internal/assets/assets.go (~L244) (path) (string, error) Streaming sha256 of a file
(*Store).SaveEvent hub/internal/store/store.go (~L1003) (...) (int64, error) Persisting ANY event (controller or hub source) Pair with dispatcher/onEvent — saving alone never notifies.
(*Store).RequestLogTail / GetPendingLogTailRequests / SaveAppLogTail hub/internal/store/logtail.go pending-intent + consume-once fulfillment THE ACK-flag pull pattern for hub→box requests (copy for any new one) SaveAppLogTail clears the request in the SAME tx (consume-once) + prunes to last 2 per (customer,app); the hub NEVER connects into a box
(*Store).RequestLogBundle / PendingLogBundleRequest / SaveLogBundle / PurgeExpiredLogBundles hub/internal/store/logbundle.go component (controller/agent) log pulls — the v0.46.0 sibling of logtail.go box-component debug-ring pulls; gzip custody, newest-3, 72 h TTL on the 60 s sweep scope = customer_id (controller/report ACK) vs host_id (agent/heartbeat envelope); SaveLogBundle runs the SECRET GATE fail-closed (blocked flag row, no payload) and clears the request in the same tx; [REDACTED]/checksums pass by design
upsertAppIssue dismissal/context semantics hub/internal/store/telemetry.go ON CONFLICT CASE guards Issue dismissal + first-capture-wins context Un-dismiss ONLY on excluded.last_seen > dismissed_at; context adopted only while stored one is empty — do not "simplify" either CASE (red-proofed)
store.GuestID hub/internal/store/store.go (~L1268) (hostID string, vmid int) string Canonical guest primary key Never hand-concatenate host+vmid.
(*Store).GetHostReportsSince + GetFirstHostReportAt + monitor.newestBackupEvidence hub/internal/store/store.go, hub/internal/monitor/deadline.go (customerID, since) ([]HostReportRow, error); (customerID) (time.Time, error); (rows, now) (time.Time, bool) Asking "when did the hub last SEE evidence of X?" instead of "what does the latest report say?" — the R-81 anchor. The agent's reporters are point-in-time and forget across a restart; the hub retains ~90 d of host-reports and does not. The three go together: window scan + first-contact anchor + a bounded lookback (backupEvidenceLookback). Never judge a report-derived absence on the LATEST report alone — that is the bug class R-81 fixed for the third time. The scan early-exits on sufficiently-fresh evidence, so don't reorder rows away from newest-first.
scheduleDaily hub/cmd/hub/main.go (~L449) (ctx, name, "HH:MM", fn, logger) Daily jobs in Europe/Budapest (prune etc.) Blocking — run as goroutine. parseHM returns 0,0 (midnight) on bad input.

2. Canonical patterns (copy structure from THE named file)

Pattern Canonical file Key traits
Monitor checker hub/internal/monitor/staleness.go Seed state on construction WITHOUT emitting events; in-memory states map under mutex; periodic Check(); emitTransition = SaveEvent then nil-checked onEvent; cleanup of vanished IDs. HostStaleness/HostDisk/HostLeaf/HostCapability/StorageFill all follow it.
API endpoint hub/internal/api/handler.go ServeHTTP (~L139) + any handler Path switch in ServeHTTP; first line of every handler = checkAuth{Customer,Host}; io.LimitReader body cap; typed anonymous payload struct; explicit 4xx strings.
Web POST action hub/internal/web/configs.go handleSetGlobalFloor (~L602) CSRF enforced centrally in web ServeHTTP; validate via normalize* helper; POST-redirect-GET with ?flash= token; log INFO on success.
Optional dependency injection hub/internal/web/server.go Set* setters (~L131148) / api handler SetDispatcher etc. Constructor takes hard deps; optional subsystems wired via SetX after construction; nil = graceful degradation (never panic).
Seam-injected sender for tests hub/internal/notify/dispatcher.go sendEmailFn (~L33) Function-field defaulting to the real impl in the constructor; tests overwrite it. Same idea: mailRateLimiter.now, mailrelay.Sender fake.
Immediate-sync notify (per plane) agent plane: poke.Notifier (web.Server.poke / api.Handler.poker via SetPoke/SetPoker) — controller plane: intent.Hub.Bump (web.Server.bumpIntent, api.Handler.intentHub) EVERY desired-state mutation fires the RIGHT plane's notifier AFTER the successful store write, never on an error path (fire-after-commit). Agent-plane pokes a HOST when its generation moved (SetHostDesired/Bump*HostDesired); controller-plane bumps a CUSTOMER on a controller-visible change. Both receivers COALESCE bursts — add NO hub-side dedup. Deliberate non-sites need a documented reason (undeliverable pre-tunnel, transport removed, or no generation bump → the 60 s ticker is the pickup path). Both seams nil-safe: unset = the ≤15-min cycle still reconciles. Full site inventory: REPORT.md audit table (v0.63.0).
Website page website/index.html UTF-8 with BOM; shared <nav>/<footer> byte-identical across pages (only class="active" differs); two-tone H1 = <h1>…<span>accent</span></h1>; all styling in website/assets/site.css tokens (:root) — zero embedded <style>; ?v=N cache-bust on site.css/icons.svg; umami snippet; no CDN fonts; no emoji (sprite icons.svg instead).
Gate script scripts/site_gates.py Byte-level mechanical gates (BOM, emoji codepoint ranges, nav/footer diff, analytics, banned tokens, cache-bust); run python scripts/site_gates.py after ANY website change; non-zero exit on failure.
Fetch-validate-install (shell) scripts/felhom-host-install.sh step_agent_install (~L1108) fetch_raw to mktemp → syntax-check (bash -n) → install -m0755 -o root -g root → only then activate; guarded-mkfs wrapper installed BEFORE the sudoers that references it (ordering is the safety property). All mutations through run() (dry-run aware).
Install-profile gate (shell) scripts/felhom-host-install.sh --mode appliance|byo (GL-2, v1.10.0) Mandatory-flag profile (no default), refusals at argv time BEFORE any prompt/step, risky step gated at its CALL SITE (one auditable place — never a branch inside the step), mode persisted to state.json + resume-mismatch refusal, FELHOM_INSTALL_STATE_DIR override for harness isolation. Harness: scripts/hostinstall-mode-harness.sh (static refusal matrix + grep-invariants + PVE dry-transcript tier; red-proofs run against a mutated scratch copy).
Disclosure↔uninstall parity (shell) scripts/felhom-host-install.sh _uninstall_statement + harness GL4-D (v1.11.0) Every host artifact the byo disclosure names must be removed OR explicitly listed KEPT by run_uninstall; the harness greps the parity (token list). New install-time artifact ⇒ add its removal + disclosure line + parity token in the SAME commit. Drive data rule: plain umount only, never -l/-f, never any format op under /mnt/felhom-drives.
Website deploy (manifest) manifests/webpage.yaml git-sync sidecar (sparse-checkout /website/ + /scripts/, --link=current) + init container waits for first sync; nginx serves current/website; push to main = deployed, no image build.
Secret handling (manifest) manifests/hub.yaml (env, ~L142) Secrets via secretKeyRef to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). report-api (the operator bearer, v0.53.0) is deliberately NOT optional: — a missing Secret fails Ready instead of booting an unauthenticatable hub. scripts/manifest_bearer_gate.py (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. ERRATA (2026-07-03): gitea-creds is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix).
Hub deploy (GitOps) manifests/hub.yaml image: (~L129) Pinned explicit tag, bumped in git, deliberate ArgoCD sync (auto-sync OFF). Code push alone deploys nothing.

3. Dangerous lookalikes — do NOT reuse

Trap Why it bites Use instead
A plain missed bool for a report-derived absence (hub/internal/monitor/deadline.go) Collapsing the three-valued verdict re-introduces one of TWO failure modes: absence→MISSED is the 2026-07-26 cry-wolf (three boxes alarmed at once, one reached a customer channel); absence→OK means a genuinely dead box alarms NEVER, which is strictly worse. Three instances of this class so far: hub v0.12.0, v0.73.0, R-81. `backupAssessment{verdict: verdictOK
(*Handler).handleNotify + formatNotificationEmail + sendResendEmail (hub/internal/api/handler.go ~L1289/1624/1589) Legacy pre-dispatcher notification trio: no cooldowns, no operator channel, no allowedEventTypes gate, duplicate Hungarian formatter. Controller path is FROZEN until slice-10 cutover. POST /api/v1/eventDispatcher.ProcessEvent + notify.Format*Email
Severity "critical" POSTed to a PRE-v0.31.0 hub Fixed in hub v0.31.0 (handleEvent now accepts critical). Older hubs coerce critical"info", which never notifies — silent alert loss. Case-variants ("Critical") still coerce on every version. Against an old hub send warning/error; otherwise lowercase critical is safe
compareVersions for anything security-ish (hub/internal/web/server.go ~L571) Returns 0 (equal) on unparseable input — a garbage version passes a floor check. gitea.compareSemver behaves differently (lexical fallback). Validate input with normalizeFloorInput first; then compareVersions is safe
Inline stringData secrets à la manifests/felhom.secret.yaml Commits real credentials to git (healthchecks superuser pw, umami APP_SECRET/POSTGRES_PASSWORD, gitea-creds admin password still live there). Out-of-band kubectl create secret + secretKeyRef (hub.yaml resend-api pattern; runbook documentation/runbooks/secrets.md)
kubectl apply / kubectl set image on manifests/ ArgoCD app felhom reverts drift on next sync; live state lies about git. Edit manifest in git → push → ArgoCD sync (CLAUDE.md steps 35)
:latest image tag in manifests Re-push doesn't change the manifest → no redeploy; Synced/Rollback misreport. Pinned version tag, bumped per deploy
grep/regex hunting emoji in website HTML Windows grep false-negatives multibyte emoji (proven in D0). python scripts/site_gates.py (codepoint-range check)
Adding a website page without touching site_gates.py PAGES list (scripts/site_gates.py ~L22) is explicit — an unlisted page is silently ungated (BOM/nav/emoji drift undetected). Add the filename to PAGES in the same commit

4. Seams & interfaces (testing + cross-repo)

Interface Defined in Implemented by Fakes/tests at
mailrelay.Sender hub/internal/mailrelay/relay.go (~L24) ResendSMTP (prod) fake sender in hub/internal/api/mail_test.go; hub/internal/mailrelay/relay_test.go
Dispatcher.sendEmailFn (func seam) hub/internal/notify/dispatcher.go (~L33) (*Dispatcher).sendEmail (Resend HTTP) hub/internal/notify/dispatcher_test.go
monitor.EventNotifyFunc hub/internal/monitor/staleness.go (~L14) closure over Dispatcher.ProcessEvent (main.go) hub/internal/monitor tests (captured-events func)
api.ConfigTemplateProvider hub/internal/api/handler.go (~L24) web.TemplateFetcher (Gitea-pulled controller.yaml template) stub providers in api tests
api.LatestVersionProvider hub/internal/api/handler.go (~L31) web.VersionChecker (registry poll) hub/internal/api/config_version_ack_test.go
mailRateLimiter.now (func seam) hub/internal/api/mail.go (~L27) time.Now hub/internal/api/mail_test.go clock injection
web.tenancyProvisioner hub/internal/web/pbsdr.go *tenantsync.Client (pinned SSH to ep0's felhom-tenantsync) fakeTenancy in hub/internal/web/pbsdr_test.go; in-process SSH server in hub/internal/tenantsync/client_test.go
Cross-repo: ep0 tenancy surface scripts/felhom-tenantsync.sh (JSON stdin/stdout forced command) installed on ep0 per runbook offsite-endpoint.md §10 provision/reissue/fingerprint ops; token secret rides stdout ONLY; the peersync script/key are untouched
Cross-repo: controller → hub POST /api/v1/report (frozen) + POST /api/v1/event felhom-controller repo new event types MUST enter allowedEventTypes (hub/internal/api/handler.go ~L1063) or the controller gets 400
Cross-repo: agent → hub POST /api/v1/host-report, /host-enroll, jobs/desired-state/escrow routes (handler.go ~L145189) felhom-agent repo hub/internal/api/host_test.go, desired_test.go, escrow_test.go, dr_test.go
Cross-repo: Day-0 bootstrap → hub GET /api/v1/config/{id} + /artifacts/{id} (X-Retrieval-Password) scripts/felhom-host-install.sh (fetches + sha256-verifies against the hub-vouched manifest) hub/internal/api/artifact_test.go
Cross-repo: controller ← hub assets GET /api/v1/assets/manifest + /assets/file/{name} felhom-controller pulls app logos/screenshots assets manifest sha-based change detection

5. Extension points (where new features plug in)

  • New event type: add to allowedEventTypes (hub/internal/api/handler.go ~L1063) and customerMessages (hub/internal/notify/templates.go) and the customer-prefs default list if customer-notifiable. Missing the first = controller POST 400s (the known gotcha).
  • New monitor checker: copy hub/internal/monitor/staleness.go (§2 pattern); wire in hub/cmd/hub/main.go with an EventNotifyFunc; severity must be warning/error/critical to notify.
  • New API route: switch in api.ServeHTTP (handler.go ~L139); auth helper first line.
  • New web page/action: switch in web.ServeHTTP (server.go ~L182) — non-GET gets CSRF automatically; template into hub/internal/web/templates/ (embedded FS, parsed in web.New); new helpers into the funcMap (server.go ~L67).
  • New template func: funcMap in web.New only; add a case to hub/internal/web/funcmap_test.go.
  • New daily job: scheduleDaily in hub/cmd/hub/main.go + add pruning to pruneAll if data grows.
  • New site gate: append to scripts/site_gates.py; new website pages go into its PAGES list.
  • New artifact kind (Day-0): consts pkg*/file* (hub/internal/web/server.go ~L27), ArtifactManifest fields + settings keys (hub/internal/store/store.go ~L905), handleSetArtifacts, artifactManifestResponse (handler.go), and the install script's verify step.
  • New host-install step: step_* function in scripts/felhom-host-install.sh using run()/fetch_raw/die helpers; keep dry-run coverage.
  • New DR-recipe section: hostHalfShape/appHalfShape and AssembledRecipe (hub/internal/store/dr_recipe.go) — those shape structs are ALLOW-LISTS, not forward-compat: a section only the emitter knows about is stored intact and silently dropped before the operator downloads it. No error, no log, no red test. That is R-122: the controller emitted offsite_restic from fork-4, all three real customers had it stored, and no delivered recipe ever contained it. Then extend TestAssembleDRRecipe_CarriesEveryEmittedSection (same commit) and, for a host-half section, the agent's DRRecipeHostHalf + BOTH copies of testdata/host-report.golden.json (byte-identical, cross-repo).

6. Known duplication (observed — NOT fixed)

  • Resend HTTP sender ×2: (*Handler).sendResendEmail (hub/internal/api/handler.go ~L1589) ≈ (*Dispatcher).sendEmail (hub/internal/notify/dispatcher.go ~L185) — byte-near-identical POST to api.resend.com. Kept because the handler copy serves the frozen legacy /notify path.
  • Hungarian customer-email formatter ×2: formatNotificationEmail (hub/internal/api/handler.go ~L1624) vs notify.FormatCustomerEmail (hub/internal/notify/templates.go ~L118). Legacy vs dispatcher; the legacy one lacks the per-event-type message map.
  • Semver compare ×2 with DIFFERENT fallback semantics: web.compareVersions (hub/internal/web/server.go ~L571, parse error → 0) vs gitea.compareSemver (hub/internal/gitea/gitea.go ~L115, parse error → lexical). Documented as deliberate (import-cycle avoidance) in gitea.go, but the behavior drift is not.
  • Checker-family structural repetition: staleness.go vs host_staleness.go, and host_disk.go vs storage_fill.go (band/bandRank/bandLabel vs bandForPercent) — same skeleton re-implemented per domain; treated as the accepted §2 pattern rather than a defect.
  • Duration formatting ×2: monitor.formatDuration (hub/internal/monitor/staleness.go ~L187) vs web.timeAgo (hub/internal/web/server.go ~L603) — different audiences (email vs UI) but overlapping logic.