felhom.eu/documentation/architecture/05-hub-architecture.md

Architecture Part 5 — The Hub

How to read this document. Two kinds of statement appear, and where this document marks them it marks them like this — the same wording as 07-backup-architecture.md:11-17, carried here on 2026-08-22 (R-376) so a reader meets one convention and not eight:

• [DESIGN] — a decision taken. Not derived from code; the code may not implement it yet.
• [FACT] — an observed property, carrying a file:line, a live command output or a citation.

Statements in this document are NOT yet all marked. Marking them wholesale is a large judgement exercise and a wrong mark is worse than none, so only what a session touches is marked (R-376). An unmarked statement therefore means "not yet classified", never "observed". That ambiguity is exactly what cost this project three sessions in August 2026: the hot/bulk placement decision sat unmarked beside a marked [FACT], and was read as an observation and reported as a defect.

Status: design draft (decision content). To be validated by Claude Code against the actual felhom-hub source (felhom.eu repo, hub/) + Parts 01–04, then placed at docs/architecture/05-hub-architecture.md.

The hub is not greenfield — it's a mature service (felhom-hub v0.11.0 as of 2026-06-13, Go + SQLite on k3s, hub.felhom.eu). This doc is the deltas to evolve it for the Proxmox model, plus the new data model. Builds on Part 1 (trust/enrollment), Part 3 (the agent + reconcile), Part 4 (signing). NOTE: this remains a design-draft for the deltas; the live hub is v0.11.0 (the prior "v0.6.3" reference was stale). Ground specifics against felhom.eu/hub/ before relying on them.

1. Source-of-truth model — two drivers, two directions

The single most important framing, and the one that governs everything below: the hub is not a monolithic source of truth. State flows in two directions with opposite drivers.

• Operator-driven intent — hub authors, agent reconciles (top-down). Which guests should exist and their spec, storage policy (a target's role/class/backup schedule), controller + golden-image versions, identity, tunnel. The operator sets these in the hub; the agent converges toward them. Here the hub is the source of truth.
• Box/customer-driven reality — box authors, pushes up, hub mirrors (bottom-up). Which USB drive is physically attached (and its durable_id), what apps are deployed and where, the customer's controller configs/settings, host/guest health, latest PBS snapshot pointers. The customer or the physical world drives these; the box reports them; the hub stays an up-to-date mirror but is never the driver.

They meet at a handshake, not a tug-of-war. Storage is the clearest case: the customer plugs in a drive → the agent detects it and reports durable_id X attached (reality) → the operator assigns role=bulk, class=slow, backup=weekly (policy, intent) → the agent reconciles that policy onto the detected drive. Apps never enter the reconcile loop — app deployment is the controller's domain (customer- or operator-driven, inside the guest); the hub only mirrors the resulting inventory. Reconciliation applies to infrastructure; the app/customer layer is mirrored.

2. Data model (Part 1 decision (b): customer-anchored)

A customer's deployment is one Host (its agent) plus one-or-more Guests (its controllers). 1 customer = 1 host + N guests; the shared-host multi-tenant case is deferred (not precluded — the hosts table is the seam it would use).

• customer_configs (existing) — the Customer anchor: identity, domain, email, retrieval_password, status, config_json. Unchanged role.
• hosts (new) — host_id PK, customer_id, api_key (the agent's hub key), agent_version, desired-state intent (storage manifest + policies + golden-image version, as JSON), a per-host desired_generation counter, the slim DR record (§9), timestamps.
• guests (new) — guest_id PK, customer_id, host_id, api_key (the controller's hub key), display_name, controller_version, per-guest desired_spec_json (CPU/mem/disk, versions), timestamps.

Per-reporter keys: today's per-customer customer_configs.api_key becomes per-reporter — hosts.api_key (agent) and guests.api_key (controller). The hub resolves a presented Bearer key → host or guest → customer; customer_configs.api_key goes unused once auth resolves via the new keys. Clean cutover: no dual-model support; the demo re-enrolls fresh into host + guests.

3. Report ingest — two domains

The single controller report splits. The de-privileged controller no longer sees host disks/storage/ backup, so its report slims (it loses System/Storage/Backup, keeps app-domain).

• POST /api/v1/host-report (new, agent) → host_reports: host CPU/RAM/disk, per-guest up/down + spec, storage-target status (attached drives + durable_id + reachability), last backup
  • restore-test per target, latest PBS snapshot pointers, cloudflared health, agent + controller versions. Denormalized columns for the dashboard; full report_json. Index (host_id, received_at DESC) + (customer_id, received_at DESC).
• POST /api/v1/report (existing, slimmed controller) → the renamed guest_reports: it gains guest_id + host_id; its cpu/memory denorm now means guest-level; backup_last_snapshot goes quiet (backup status lives in host_reports). App telemetry / log issues stay.

These two streams are the bottom-up mirror of §1 — they keep the hub current without a separate push.

4. Liveness / dead-man's-switch

Evolves the existing staleness checker (60s cadence, a configured threshold — 30 m until 2026-09-17, 45 m since operator ruling A on R-549; OK under it, down at 2× = 90 m; today: controller-report recency → node_stale/down/recovered):

• Primary = host-report recency → host_stale / host_down. The agent heartbeat is the box's liveness signal; a silent agent = the box is gone (the critical alert).
• Guest up/down comes from the host report's per-guest status — authoritative, every poll, faster than waiting for a guest report to go stale.
• Guest-report recency = secondary app-level signal.

Backup-deadline checker: today it is event-based — it scans for backup_completed/backup_failed events since local midnight and alerts if none. Two changes: (1) mechanism — move it to a field check on host_reports' last-backup-per-target (cleaner now that backup state arrives in the host report); (2) emitter — the de-privileged controller no longer runs backups, so the agent is the source of the last-backup status (Part 3 §8). Without re-homing the source, the deadline check would go silent after the controller stops backing up.

5. Desired-state serving

The operator's intent (§1 top-down) lives as JSON on hosts/guests (storage manifest + policies + golden version on the host; per-guest spec + versions on the guest) with a per-host desired_generation. The agent pulls its host's desired state on poll (with the generation, so it reconciles only on change and reports which generation it has converged to).

• Benign convergence (create a guest, attach storage per policy, bump a version, adjust a non-destructive policy) → the agent reconciles freely.
• Destructive convergence (guest removal = destroy, storage detach/wipe, data-losing resize) → the agent requires a matching signed op (§6) before executing that delta; absent/invalid → it refuses and reports pending_signature.

Geo is not in the agent's desired state — it's customer→hub→Cloudflare (§7); the agent never touches WAF.

The controller floor and its agent requirement (hub v0.112.0, R-472). The managed controller floor rides the report ACK. store.ResolveManagedFloor decides per box whether to serve it, from three inputs: the floor in force (per-customer override, else global), the vouched Day-0 manifest (golden version + MinAgent), and a declared MinAgent stored beside the floor (hub_settings.min_controller_version_declared_min_agent for the global floor, customer_configs.min_controller_declared_min_agent for an override, each as FLOOR=MINAGENT so it binds only to the floor it was saved with). Inside the golden the manifest's MinAgent governs. Above the golden the declared one does; with no declaration the floor is held beyond the golden, as since R-216. Either way the box's reported agent must meet the chosen MinAgent, else the floor is held with agent <v> < MinAgent <w>. The decision carries MinAgentSource (manifest / declared), shown on the Hosts page and logged once per change as managed floor SERVED. The rules the operator follows: runbooks/publish-train-rules.md rule 1.

6. Authorization — signed-op queue + editing flow

Implements Part 4's gate on the hub side. The hub holds no signing key.

• signed_ops (new): op_id, customer_id, host_id, target_guest, op_type, op_blob (canonical JSON), signature (armored SSHSIG), status (pending_signature → signed → delivered → executed / failed / expired / rejected), nonce, issued_at, expires_at, executed_at, result.
• Editing flow: the operator edits a customer's desired state, reusing the existing config-form + diff UX. Note the transport inverts: today's "Push" is a hub→box inbound POST (forbidden by the box-initiated model); here "publish" means write to desired state, delivered on the next agent/ controller poll. The form and diff carry over; the push transport does not. The hub diffs vs current and classifies each delta (B1 rule):
  • benign → published straight to desired state;
  • destructive → the hub generates the canonical op blob and routes it through signing.
• Signing hand-off (Part 4 option (b)): a local operator CLI (felhom-sign --pending) fetches the pending blob from the hub, signs it on the workstation with the dedicated key, and posts the signature back into signed_ops. The hub never sees the key.
• The agent polls signed_ops for its host alongside desired state, verifies (Part 4 pipeline), executes, and reports status → the hub logs to the existing events audit trail.
• Classification lives in both places, with different jobs: the hub classifies at edit time for UX (prompt to sign); the agent's classification is the authoritative guard (a compromised hub could skip the prompt, but the agent still enforces the signature).
• A pending-ops view per customer shows the lifecycle (awaiting signature → awaiting agent → executed).

7. Geo enforcement (Part-2 S4)

The hub already holds the CF API token and already has a remove-all path (internal/web/configs.go handleGeoDisable → cloudflare.RemoveGeoRules). But the token is dual-purpose today — DNS-01/ACME and WAF/geo — and configgen.Generate deep-merges it (via config_json) into the generated controller.yaml, so it currently ships down to the box. Two things follow:

• ACME assumption (must be stated, not skipped): in the Cloudflare-Tunnel-default model the edge terminates TLS, so the box needs no public certificate and the DNS-01/ACME use of the token goes away. Granting that, the token comes fully off the box and lives hub-only. (If any box still does DNS-01, the token cannot fully come off — so this assumption is load-bearing.)
• configgen must stop emitting cf_api_token into controller.yaml (drop it from the merge / relocate it to a hub-only field).

The delta: the customer sets geo in the controller UI → the controller reports the geo desired-state up → the hub reconciles it into the Cloudflare WAF (rather than the box calling the CF API). The hub keeps the remove-all override for self-lockout. The controller no longer calls the CF API.

8. Enrollment (evolution of the existing retrieval-password/config-gen flow)

Today: GET /config/{id} with an X-Retrieval-Password (Hungarian passphrase) returns a deep-merged controller.yaml. New:

• Enrollment mints the agent identity first (the agent then provisions controllers), pins the operator signing public keys (Part 4 — operational + cold recovery) onto the agent, and the agent mints each controller's bootstrap (its hub guest key + local-API token).
• A restore-mode re-enrollment (§9) hands an existing identity to a fresh agent.

The existing configgen deep-merge + Hungarian-passphrase machinery is the base; it grows the agent-first + key-pinning + restore-mode steps.

9. DR model

The headline: the old heavy infra-backup push retires — not because the hub authors everything (§1 says it doesn't), but because (a) the box-driven mirror already arrives via the §3 report streams, and (b) the actual app data + configs live inside the PBS guest snapshot. So a separate config+secrets+restic-password infra-backup blob is redundant.

What remains:

• the report streams keep the hub's mirror current (storage layout + durable_ids, app inventory, snapshot pointers) — but this mirror is convenience, not the DR source of record (reports are pruned by age);
• the agent escrows the recovery-code-wrapped PBS key to the hub (the one artifact only the box can produce — zero-knowledge: the hub stores it, cannot open it);
• a slim DR record on the hosts row (PBS namespace + repo fingerprint + the wrapped escrow key). These last two are box-reported columns on an otherwise operator-intent row — labelled as such so the §1 two-driver split stays legible per column.

Both existing infra-backup tables retire — infra_backup_versions (the current/live one, all readers hit it) and infra_backups (the deprecated legacy mirror). The slim DR record folds onto hosts instead. The controller's infra-backup push is removed (it's de-privileged).

Recovery (host loss): the new agent re-enrolls in restore mode; the hub hands it the durable record — and DR reads from the durable sources, not the prunable report mirror: operator intent (desired-state on hosts/guests — identity, tunnel token, storage manifest), the slim DR record (PBS namespace + repo fingerprint), the wrapped escrow key, and PBS's own snapshot enumeration (the agent lists snapshots once it has the namespace + unwrapped key). Guest inventory + app data come from inside the PBS guest snapshots, not from a retained host_report, so recovery doesn't degrade when the last report has aged out. The customer provides their recovery code at the agent, which unwraps the PBS key locally (never sent to the hub); the agent restores guests from PBS, resets identity, reuses the tunnel. The customer recovery code is the irreducible residual (the premium operator-managed custody tier avoids it, at the cost of the operator holding the key). The old controller-targeted GET /recovery/{id} is replaced by this agent restore-mode flow.

10. What persists from today (unchanged or lightly adapted)

The Customer record (customer_configs); config generation/retrieval (configgen); the two-tier notification system (operator English / customer Hungarian, Resend, cooldowns); events + audit; app_telemetry / app_log_issues; customer lifecycle actions (block/unblock, trigger-update, delete); the asset manager; and the dashboard — adapted to render the host + guests view per customer instead of a single controller.

11. Schema deltas (grounded in store.go's idempotent style; clean cutover)

• NEW: hosts, guests, host_reports, signed_ops.
• DROP reports + CREATE guest_reports (under the clean cutover this is drop+create with no data migration, not an in-place rename); guest_reports adds guest_id, host_id; cpu/memory mean guest-level; backup_last_snapshot goes quiet.
• ADD desired-state JSON + desired_generation to hosts; desired_spec_json to guests; the slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto hosts.
• DROP both infra_backup_versions (current/live) and infra_backups (legacy mirror) — the DR record replaces them on hosts.
• KEEP customer_configs, events, customer_notifications, notification_log, app_telemetry, app_log_issues.
• Authz cleanup the cutover enables: several endpoints today use global-or-any-customer-key auth rather than customer-scoped (the infra-backup GETs, /notify). Most retire with the infra-backup push; any that carry over should scope to the resolved host/guest → customer under §2.

12. Open items

• Operator signing-key operational mechanics (Part 4 §8) — the hub-side pending-op UI is here; the key custody/rotation tooling is Part 4's.
• Multi-tenant resource fairness (deferred shared-host case).
• Hub-side desired-state editing UX specifics (form/diff wiring) — to be grounded against hub/internal/web/configs.go at implementation.
• Golden-image refresh cadence / fleet versioning (carried from Part 3 §13).

13. When the self-bind (connect) link is sent [DESIGN, operator decision A 2026-09-15, hub v0.114.0]

The box's console tells the volunteer to open „az e-mailben kapott link". That mail must exist whenever a customer is waiting for a box, without an operator press. autoMintSelfBindLink runs on four events: customer creation; RESET completion; an e-mail set or changed on a customer with no bound host (config save); a host delete that keeps the customer. The last two re-check that no host is bound, so a customer who has a box is never sent a pairing link. Appliance registration is not a trigger — it knows no customer (api/appliance.go). Every send is stored as a hub-internal selfbind_link_sent event with its occasion; the Setup tab shows „Kapcsolódó link elküldve: ()" and keeps the button as the manual resend. Until 2026-09-15 only the first two existed, and BIGNIGHT's tester-1 never got its mail (R-509).

14. The PBS-DR descriptor and its endpoint token — lifecycle [DESIGN, recorded 2026-09-15]

Two records must agree: the token on the endpoint (ep0: namespace <customer> + token felhom@pbs!<customer>) and the descriptor in the host's desired_json (pbs_dr: namespace, token id, fingerprint, datastore, tunnel IP, secret_generation) with its consume-once secret.

event	token on ep0	descriptor on hub
DR tier ON + WG peer present (form save or WG hook)	provision	created, secret stored
re-issue, descriptor present	re-keyed (delete + recreate)	secret_generation bumped
re-issue, descriptor ABSENT, DR flag on (R-511, v0.114.0)	adopted: re-keyed	rebuilt from the endpoint's answer; pbsdr_adopted audit row
host delete	kept (tenancy survives)	goes with the host
host delete acknowledged through escrow-ack, then a new box	re-keyed automatically (F-14)	rebuilt
RESET / customer delete	deprovisioned — namespace, every backup group AND token	purged

Why adopt exists: a rebuilt box's WG hook refused („the endpoint already holds a PBS token … use the explicit Re-issue action") and the re-issue itself then refused with 400 — no button restored the tier. Not built: releasing ONLY the token on host delete. The endpoint's only removal op (deprovision) destroys the backups too, so a token-only release needs a new endpoint operation.

15. Customer e-mails — what the hub writes to a household [hub v0.118.0, R-558]

The section this document did not have. The hub composes every sentence a household reads before it has seen any box screen, and until v0.118.0 nothing here described that.

15.1 The four mails

Mail	Trigger	Rendered by	Language source
Event notification (39 event types)	a box event, or a hub checker	FormatCustomerEmail	reported → created-with → hu
Claim / reset / re-enroll / claimed	the claim arc	FormatClaimEmail	created-with (no box has reported yet)
Self-bind link	customer creation, or the operator's button	FormatSelfBindEmail	created-with
The public bind PAGE at /bind/<token>	the customer opens the link	one template per language	created-with, except expired — see 15.4

The operator's channel (FormatOperatorEmail, and the R-182 backup-run digest) is not in this table and is not localised. It is English, it names host ids and blob counts, and it is untouched.

15.2 Where the sentences live

hub/internal/i18n/locales/{hu,en}.json, one flat key→text map per language. Hungarian is authoritative and holds every key; a key missing from English renders the Hungarian and is counted by a gate held at zero. customerMessages and severityLabels are DERIVED from the bundle rather than being literals, so a sentence is written in exactly one place. A new event type therefore needs a line in hu.json and its English twin, alongside its allowedEventTypes entry — the long-standing "both together" rule, in its new home.

15.3 The language order, and why it is that order

Last reported → created-with → Hungarian (Store.CustomerLanguage).

1. What the box last reported is what the HOUSEHOLD chose on their own dashboard. It outranks everything else: the operator's creation-time pick is a default, never an override.
2. The creation-time language (customer_configs.language) covers the window before any box has reported — which is precisely when the claim mail and the bind page are sent, so it is not an edge case. It also seeds the box: configgen writes it as customer.language.
3. Hungarian, for every customer that predates all of this.

Two storage rules follow from that order and are easy to get wrong:

• reports.language defaults to empty, never hu. Empty means this box has never told us, which is not the same as this household chose Hungarian — a controller older than v0.247.0 sends no language at all, and storing hu would make a later real choice indistinguishable from the absence of one.
• The newest report is found by the autoincrement id, not by received_at. received_at has second granularity, so two reports arriving in one second tie and the winner is arbitrary.

A quiet-box alarm deliberately uses the last REPORTED language even though the box is silent: the last thing it said is still the best thing known about the household.

15.4 The box's own sentences, and the one thing the hub cannot do

About a third of the customer mails carry a sentence the BOX composed, naming a drive, an app or a number. The hub cannot translate one. So the box sends the household's version beside the Hungarian one, as message_customer on POST /api/v1/event; the hub puts that in the household's mail and keeps the Hungarian for the operator's. It is additive and optional forever — a parked box will never send it, and its absence must leave the mail exactly as it was.

Until every box runs controller v0.256.0 or later, an English household's mail can carry one Hungarian line. The rest of the mail is English. That is expected, not a defect.

The bind page is a no-oracle surface, and the LANGUAGE is part of that. The page folds an unknown token into expired so a stranger cannot learn whether a link was ever real. If it then rendered a real English customer's expired token in English and an unknown one in Hungarian, the language would answer the question the text refuses to — for every customer who is not Hungarian. The expired state therefore always renders in the default language; every other state already discloses that the token is real. Pinned by TestBindExpiredIsAlwaysDefaultLanguage.

15.5 How "the Hungarian did not change" is known

56 goldens captured from v0.117.0 before any string moved, in hub/internal/notify/testdata/mail_goldens/hu/, with the English set beside them. The claim is a diff, not a reading. A golden is never regenerated to make a change pass.