felhom.eu/documentation/architecture/05-hub-architecture.md

# Architecture Part 5 — The Hub

> **How to read this document.** Two kinds of statement appear, and where this document marks them it
> marks them like this — the same wording as `07-backup-architecture.md:11-17`, carried here on
> 2026-08-22 (R-376) so a reader meets one convention and not eight:
>
> - **[DESIGN]** — a decision taken. Not derived from code; the code may not implement it yet.
> - **[FACT]** — an observed property, carrying a `file:line`, a live command output or a citation.
>
> **Statements in this document are NOT yet all marked.** Marking them wholesale is a large judgement
> exercise and a wrong mark is worse than none, so only what a session touches is marked (R-376).
> **An unmarked statement therefore means "not yet classified", never "observed".** That ambiguity is
> exactly what cost this project three sessions in August 2026: the hot/bulk placement decision sat
> unmarked beside a marked `[FACT]`, and was read as an observation and reported as a defect.



> Status: design draft (decision content). To be validated by Claude Code against the **actual
> felhom-hub source** (`felhom.eu` repo, `hub/`) + Parts 01–04, then placed at
> `docs/architecture/05-hub-architecture.md`.
>
> The hub is **not** greenfield — it's a mature service (felhom-hub **v0.11.0** as of 2026-06-13, Go +
> SQLite on k3s, `hub.felhom.eu`). This doc is the **deltas** to evolve it for the Proxmox model, plus
> the new data model. Builds on Part 1 (trust/enrollment), Part 3 (the agent + reconcile), Part 4 (signing).
> NOTE: this remains a design-draft for the deltas; the live hub is v0.11.0 (the prior "v0.6.3" reference
> was stale). Ground specifics against `felhom.eu/hub/` before relying on them.

## 1. Source-of-truth model — two drivers, two directions

The single most important framing, and the one that governs everything below: the hub is **not** a
monolithic source of truth. State flows in two directions with opposite drivers.

- **Operator-driven *intent* — hub authors, agent reconciles (top-down).** Which guests should
  exist and their spec, storage *policy* (a target's role/class/backup schedule), controller +
  golden-image versions, identity, tunnel. The operator sets these in the hub; the agent converges
  toward them. Here the hub *is* the source of truth.
- **Box/customer-driven *reality* — box authors, pushes up, hub mirrors (bottom-up).** Which USB
  drive is *physically* attached (and its `durable_id`), what apps are deployed and where, the
  customer's controller configs/settings, host/guest health, latest PBS snapshot pointers. The
  customer or the physical world drives these; the box reports them; the hub stays an up-to-date
  **mirror** but is **never** the driver.

They meet at a **handshake**, not a tug-of-war. Storage is the clearest case: the customer plugs in
a drive → the agent *detects* it and reports `durable_id X attached` (reality) → the operator
assigns `role=bulk, class=slow, backup=weekly` (policy, intent) → the agent reconciles that policy
*onto the detected drive*. **Apps never enter the reconcile loop** — app deployment is the
controller's domain (customer- or operator-driven, inside the guest); the hub only mirrors the
resulting inventory. **Reconciliation applies to infrastructure; the app/customer layer is mirrored.**

## 2. Data model (Part 1 decision (b): customer-anchored)

A customer's deployment is one **Host** (its agent) plus one-or-more **Guests** (its controllers).
1 customer = 1 host + N guests; the shared-host multi-tenant case is deferred (not precluded — the
`hosts` table is the seam it would use).

- **`customer_configs`** (existing) — the Customer anchor: identity, domain, email,
  `retrieval_password`, status, config_json. Unchanged role.
- **`hosts`** (new) — `host_id PK, customer_id, api_key` (the agent's hub key), `agent_version`,
  desired-state intent (storage manifest + policies + golden-image version, as JSON), a per-host
  **`desired_generation`** counter, the slim DR record (§9), timestamps.
- **`guests`** (new) — `guest_id PK, customer_id, host_id, api_key` (the controller's hub key),
  `display_name, controller_version`, per-guest **`desired_spec_json`** (CPU/mem/disk, versions),
  timestamps.

**Per-reporter keys:** today's per-customer `customer_configs.api_key` becomes per-reporter —
`hosts.api_key` (agent) and `guests.api_key` (controller). The hub resolves a presented Bearer key →
host or guest → customer; `customer_configs.api_key` goes unused once auth resolves via the new keys.
**Clean cutover:** no dual-model support; the demo re-enrolls fresh into `host + guests`.

## 3. Report ingest — two domains

The single controller report splits. The de-privileged controller no longer sees host disks/storage/
backup, so its report **slims** (it loses System/Storage/Backup, keeps app-domain).

- **`POST /api/v1/host-report`** (new, agent) → **`host_reports`**: host CPU/RAM/disk, per-guest
  up/down + spec, storage-target status (attached drives + `durable_id` + reachability), last backup
  + restore-test per target, latest PBS snapshot pointers, `cloudflared` health, agent + controller
  versions. Denormalized columns for the dashboard; full `report_json`. Index `(host_id, received_at
  DESC)` + `(customer_id, received_at DESC)`.
- **`POST /api/v1/report`** (existing, slimmed controller) → the renamed **`guest_reports`**: it
  gains `guest_id` + `host_id`; its `cpu/memory` denorm now means *guest-level*; `backup_last_snapshot`
  goes quiet (backup status lives in `host_reports`). App telemetry / log issues stay.

These two streams are the bottom-up mirror of §1 — they keep the hub current without a separate push.

## 4. Liveness / dead-man's-switch

Evolves the existing staleness checker (60s **cadence**, a **configured** threshold — 30 m until
2026-09-17, **45 m** since operator ruling A on R-549; OK under it, down at 2× = 90 m; today: controller-report recency → `node_stale`/`down`/`recovered`):

- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
  liveness signal; a silent agent = the box is gone (the critical alert).
- **Guest up/down comes from the host report's per-guest status** — authoritative, every poll, faster
  than waiting for a guest report to go stale.
- **Guest-report recency = secondary** app-level signal.

**Backup-deadline checker:** today it is *event-based* — it scans for `backup_completed`/`backup_failed`
events since local midnight and alerts if none. Two changes: (1) **mechanism** — move it to a field
check on `host_reports`' last-backup-per-target (cleaner now that backup state arrives in the host
report); (2) **emitter** — the de-privileged controller no longer runs backups, so the **agent** is the
source of the last-backup status (Part 3 §8). Without re-homing the source, the deadline check would go
silent after the controller stops backing up.

## 5. Desired-state serving

The operator's **intent** (§1 top-down) lives as JSON on `hosts`/`guests` (storage manifest +
policies + golden version on the host; per-guest spec + versions on the guest) with a per-host
`desired_generation`. The agent pulls its host's desired state on poll (with the generation, so it
reconciles only on change and reports which generation it has converged to).

- **Benign convergence** (create a guest, attach storage per policy, bump a version, adjust a
  non-destructive policy) → the agent reconciles freely.
- **Destructive convergence** (guest removal = destroy, storage detach/wipe, data-losing resize) →
  the agent requires a **matching signed op** (§6) before executing that delta; absent/invalid → it
  refuses and reports `pending_signature`.

**Geo is *not* in the agent's desired state** — it's customer→hub→Cloudflare (§7); the agent never
touches WAF.

**The controller floor and its agent requirement (hub v0.112.0, R-472).** The managed controller floor
rides the report ACK. `store.ResolveManagedFloor` decides per box whether to serve it, from three
inputs: the floor in force (per-customer override, else global), the vouched Day-0 manifest (golden
version + MinAgent), and a **declared MinAgent** stored beside the floor
(`hub_settings.min_controller_version_declared_min_agent` for the global floor,
`customer_configs.min_controller_declared_min_agent` for an override, each as `FLOOR=MINAGENT` so it
binds only to the floor it was saved with). Inside the golden the manifest's MinAgent governs. Above
the golden the declared one does; with no declaration the floor is **held beyond the golden**, as since
R-216. Either way the box's reported agent must meet the chosen MinAgent, else the floor is held with
`agent <v> < MinAgent <w>`. The decision carries `MinAgentSource` (`manifest` / `declared`), shown on
the Hosts page and logged once per change as `managed floor SERVED`. The rules the operator follows:
`runbooks/publish-train-rules.md` rule 1.

## 6. Authorization — signed-op queue + editing flow

Implements Part 4's gate on the hub side. The hub holds **no signing key**.

- **`signed_ops`** (new): `op_id, customer_id, host_id, target_guest, op_type, op_blob (canonical
  JSON), signature (armored SSHSIG), status (pending_signature → signed → delivered → executed /
  failed / expired / rejected), nonce, issued_at, expires_at, executed_at, result`.
- **Editing flow:** the operator edits a customer's desired state, reusing the existing config-form +
  diff UX. Note the **transport inverts**: today's "Push" is a hub→box *inbound* POST (forbidden by the
  box-initiated model); here "publish" means **write to desired state, delivered on the next agent/
  controller poll**. The form and diff carry over; the push transport does not. The hub diffs vs current
  and **classifies each delta** (B1 rule):
  - **benign** → published straight to desired state;
  - **destructive** → the hub generates the canonical op blob and routes it through signing.
- **Signing hand-off (Part 4 option (b)):** a local operator CLI (`felhom-sign --pending`) fetches
  the pending blob from the hub, signs it on the workstation with the dedicated key, and posts the
  signature back into `signed_ops`. The hub never sees the key.
- The agent polls `signed_ops` for its host alongside desired state, verifies (Part 4 pipeline),
  executes, and reports status → the hub logs to the existing **`events`** audit trail.
- **Classification lives in both places, with different jobs:** the hub classifies at *edit time*
  for UX (prompt to sign); the **agent's classification is the authoritative guard** (a compromised
  hub could skip the prompt, but the agent still enforces the signature).
- A **pending-ops view** per customer shows the lifecycle (awaiting signature → awaiting agent →
  executed).

## 7. Geo enforcement (Part-2 S4)

The hub already holds the CF API token and already has a remove-all path
(`internal/web/configs.go` `handleGeoDisable` → `cloudflare.RemoveGeoRules`). **But the token is
dual-purpose today** — DNS-01/ACME *and* WAF/geo — and `configgen.Generate` deep-merges it (via
`config_json`) into the generated `controller.yaml`, so it currently ships **down to the box**. Two
things follow:

- **ACME assumption (must be stated, not skipped):** in the Cloudflare-Tunnel-default model the edge
  terminates TLS, so the box needs no public certificate and the **DNS-01/ACME use of the token goes
  away**. Granting that, the token comes fully off the box and lives hub-only. (If any box still does
  DNS-01, the token cannot fully come off — so this assumption is load-bearing.)
- **`configgen` must stop emitting `cf_api_token`** into `controller.yaml` (drop it from the merge /
  relocate it to a hub-only field).

The delta: the **customer sets geo in the controller UI → the controller reports the geo desired-state
up → the hub reconciles it into the Cloudflare WAF** (rather than the box calling the CF API). The hub
keeps the remove-all override for self-lockout. The controller no longer calls the CF API.

## 8. Enrollment (evolution of the existing retrieval-password/config-gen flow)

Today: `GET /config/{id}` with an `X-Retrieval-Password` (Hungarian passphrase) returns a deep-merged
`controller.yaml`. New:

- Enrollment mints the **agent identity first** (the agent then provisions controllers), pins the
  **operator signing public keys** (Part 4 — operational + cold recovery) onto the agent, and the
  agent mints each controller's bootstrap (its hub guest key + local-API token).
- A **restore-mode** re-enrollment (§9) hands an existing identity to a fresh agent.

The existing `configgen` deep-merge + Hungarian-passphrase machinery is the base; it grows the
agent-first + key-pinning + restore-mode steps.

## 9. DR model

The headline: the **old heavy infra-backup push retires** — not because the hub authors everything
(§1 says it doesn't), but because (a) the box-driven mirror already arrives via the §3 report streams,
and (b) the actual app **data + configs live inside the PBS guest snapshot**. So a separate
config+secrets+restic-password infra-backup blob is redundant.

What remains:
- the **report streams** keep the hub's mirror current (storage layout + `durable_id`s, app inventory,
  snapshot pointers) — but this mirror is **convenience, not the DR source of record** (reports are
  pruned by age);
- the agent **escrows the recovery-code-wrapped PBS key** to the hub (the one artifact only the box
  can produce — zero-knowledge: the hub stores it, cannot open it);
- a **slim DR record** on the `hosts` row (PBS namespace + repo fingerprint + the wrapped escrow key).
  These last two are *box-reported* columns on an otherwise operator-intent row — labelled as such so
  the §1 two-driver split stays legible per column.

Both existing infra-backup tables retire — `infra_backup_versions` (the current/live one, all readers
hit it) **and** `infra_backups` (the deprecated legacy mirror). The slim DR record folds onto `hosts`
instead. The **controller's infra-backup push is removed** (it's de-privileged).

**Recovery (host loss):** the new agent re-enrolls in **restore mode**; the hub hands it the durable
record — and DR reads from the **durable sources, not the prunable report mirror**: operator intent
(desired-state on `hosts`/`guests` — identity, tunnel token, storage manifest), the slim DR record
(PBS namespace + repo fingerprint), the **wrapped escrow key**, and **PBS's own snapshot enumeration**
(the agent lists snapshots once it has the namespace + unwrapped key). Guest inventory + app data come
from **inside the PBS guest snapshots**, not from a retained `host_report`, so recovery doesn't degrade
when the last report has aged out. The **customer provides their recovery code at the agent**, which
unwraps the PBS key locally (never sent to the hub); the agent restores guests from PBS, resets
identity, reuses the tunnel. The customer recovery code is the irreducible residual (the premium
operator-managed custody tier avoids it, at the cost of the operator holding the key). The old
controller-targeted `GET /recovery/{id}` is replaced by this agent restore-mode flow.

## 10. What persists from today (unchanged or lightly adapted)

The Customer record (`customer_configs`); config generation/retrieval (`configgen`); the two-tier
notification system (operator English / customer Hungarian, Resend, cooldowns); `events` + audit;
`app_telemetry` / `app_log_issues`; customer lifecycle actions (block/unblock, trigger-update,
delete); the asset manager; and the dashboard — adapted to render the **host + guests** view per
customer instead of a single controller.

## 11. Schema deltas (grounded in store.go's idempotent style; clean cutover)

- **NEW:** `hosts`, `guests`, `host_reports`, `signed_ops`.
- **DROP `reports` + CREATE `guest_reports`** (under the clean cutover this is drop+create with no data
  migration, not an in-place rename); `guest_reports` adds `guest_id`, `host_id`; `cpu/memory` mean
  guest-level; `backup_last_snapshot` goes quiet.
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`; the
  slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`.
- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR
  record replaces them on `hosts`.
- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`,
  `app_telemetry`, `app_log_issues`.
- **Authz cleanup the cutover enables:** several endpoints today use global-or-any-customer-key auth
  rather than customer-scoped (the infra-backup GETs, `/notify`). Most retire with the infra-backup
  push; any that carry over should scope to the resolved host/guest → customer under §2.

## 12. Open items
- Operator signing-key operational mechanics (Part 4 §8) — the hub-side pending-op UI is here; the
  key custody/rotation tooling is Part 4's.
- Multi-tenant resource fairness (deferred shared-host case).
- Hub-side desired-state **editing UX** specifics (form/diff wiring) — to be grounded against
  `hub/internal/web/configs.go` at implementation.
- Golden-image refresh cadence / fleet versioning (carried from Part 3 §13).

## 13. When the self-bind (connect) link is sent [DESIGN, operator decision A 2026-09-15, hub v0.114.0]

The box's console tells the volunteer to open „az e-mailben kapott link". That mail must exist whenever
a customer is **waiting for a box**, without an operator press. `autoMintSelfBindLink` runs on four
events: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
bound host** (config save); **a host delete** that keeps the customer. The last two re-check that no
host is bound, so a customer who has a box is never sent a pairing link. **Appliance registration is
not a trigger** — it knows no customer (`api/appliance.go`). Every send is stored as a hub-internal
`selfbind_link_sent` event with its occasion; the Setup tab shows „Kapcsolódó link elküldve: <date>
(<occasion>)" and keeps the button as the manual resend. Until 2026-09-15 only the first two existed,
and BIGNIGHT's tester-1 never got its mail (R-509).

## 14. The PBS-DR descriptor and its endpoint token — lifecycle [DESIGN, recorded 2026-09-15]

Two records must agree: the **token** on the endpoint (ep0: namespace `<customer>` + token
`felhom@pbs!<customer>`) and the **descriptor** in the host's `desired_json` (`pbs_dr`: namespace,
token id, fingerprint, datastore, tunnel IP, `secret_generation`) with its consume-once secret.

| event | token on ep0 | descriptor on hub |
|---|---|---|
| DR tier ON + WG peer present (form save or WG hook) | **provision** | created, secret stored |
| re-issue, descriptor present | **re-keyed** (delete + recreate) | `secret_generation` bumped |
| **re-issue, descriptor ABSENT, DR flag on** (R-511, v0.114.0) | **adopted**: re-keyed | **rebuilt** from the endpoint's answer; `pbsdr_adopted` audit row |
| host delete | **kept** (tenancy survives) | goes with the host |
| host delete acknowledged through escrow-ack, then a new box | re-keyed automatically (F-14) | rebuilt |
| RESET / customer delete | **deprovisioned** — namespace, every backup group AND token | purged |

**Why adopt exists:** a rebuilt box's WG hook refused („the endpoint already holds a PBS token … use
the explicit Re-issue action") and the re-issue itself then refused with 400 — no button restored the
tier. **Not built:** releasing ONLY the token on host delete. The endpoint's only removal op
(`deprovision`) destroys the backups too, so a token-only release needs a new endpoint operation.

## 15. Customer e-mails — what the hub writes to a household [hub v0.118.0, R-558]

**The section this document did not have.** The hub composes every sentence a household reads before
it has seen any box screen, and until v0.118.0 nothing here described that.

### 15.1 The four mails

| Mail | Trigger | Rendered by | Language source |
|---|---|---|---|
| Event notification (39 event types) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
| Claim / reset / re-enroll / claimed | the claim arc | `FormatClaimEmail` | created-with (no box has reported yet) |
| Self-bind link | customer creation, or the operator's button | `FormatSelfBindEmail` | created-with |
| The public bind PAGE at `/bind/<token>` | the customer opens the link | one template per language | created-with, **except `expired`** — see 15.4 |

The operator's channel (`FormatOperatorEmail`, and the R-182 backup-run digest) is **not** in this
table and is not localised. It is English, it names host ids and blob counts, and it is untouched.

### 15.2 Where the sentences live

`hub/internal/i18n/locales/{hu,en}.json`, one flat key→text map per language. Hungarian is
authoritative and holds every key; a key missing from English renders the Hungarian and is counted by
a gate held at zero. `customerMessages` and `severityLabels` are DERIVED from the bundle rather than
being literals, so a sentence is written in exactly one place. **A new event type therefore needs a
line in `hu.json` and its English twin**, alongside its `allowedEventTypes` entry — the long-standing
"both together" rule, in its new home.

### 15.3 The language order, and why it is that order

**Last reported → created-with → Hungarian** (`Store.CustomerLanguage`).

1. **What the box last reported** is what the HOUSEHOLD chose on their own dashboard. It outranks
   everything else: the operator's creation-time pick is a default, never an override.
2. **The creation-time language** (`customer_configs.language`) covers the window before any box has
   reported — which is precisely when the claim mail and the bind page are sent, so it is not an edge
   case. It also seeds the box: configgen writes it as `customer.language`.
3. **Hungarian**, for every customer that predates all of this.

Two storage rules follow from that order and are easy to get wrong:

- `reports.language` defaults to **empty**, never `hu`. Empty means *this box has never told us*,
  which is not the same as *this household chose Hungarian* — a controller older than v0.247.0 sends
  no language at all, and storing `hu` would make a later real choice indistinguishable from the
  absence of one.
- The newest report is found by the autoincrement **`id`**, not by `received_at`. `received_at` has
  second granularity, so two reports arriving in one second tie and the winner is arbitrary.

A quiet-box alarm deliberately uses the last REPORTED language even though the box is silent: the
last thing it said is still the best thing known about the household.

### 15.4 The box's own sentences, and the one thing the hub cannot do

About a third of the customer mails carry a sentence the BOX composed, naming a drive, an app or a
number. **The hub cannot translate one.** So the box sends the household's version beside the
Hungarian one, as `message_customer` on `POST /api/v1/event`; the hub puts that in the household's
mail and keeps the Hungarian for the operator's. It is additive and optional **forever** — a parked
box will never send it, and its absence must leave the mail exactly as it was.

Until every box runs controller v0.256.0 or later, an English household's mail can carry one
Hungarian line. The rest of the mail is English. That is expected, not a defect.

**The bind page is a no-oracle surface, and the LANGUAGE is part of that.** The page folds an unknown
token into `expired` so a stranger cannot learn whether a link was ever real. If it then rendered a
real English customer's expired token in English and an unknown one in Hungarian, the language would
answer the question the text refuses to — for every customer who is not Hungarian. The `expired`
state therefore always renders in the default language; every other state already discloses that the
token is real. Pinned by `TestBindExpiredIsAlwaysDefaultLanguage`.

### 15.5 How "the Hungarian did not change" is known

56 goldens captured from v0.117.0 before any string moved, in
`hub/internal/notify/testdata/mail_goldens/hu/`, with the English set beside them. The claim is a
diff, not a reading. A golden is never regenerated to make a change pass.

### 15.6 The CODES a household types, per language [hub v0.119.0, R-597]

A mail in English that carries three Hungarian words with accents is not an English mail. The 2026-09-20
drill received exactly that, and could paste the code but not read it to anyone.

**Two secrets this repo mints follow the household's language**, list and word count chosen together
by `configgen.RandomPassphraseFor(lang, use)`:

| Secret | hu | en | who calls it |
|---|---|---|---|
| Setup / reset code | 3 words, 44.6 bits | **4 words, 51.7 bits** | `claim.Engine` via `CustomerLanguage` |
| Owner passphrase | 5 words, 74.3 bits | **6 words, 77.5 bits** | the three `configs.go` sites |

**The rule is per use: English ≥ Hungarian, in bits.** The English list (EFF large, 7772 words after
filtering) carries 12.92 bits/word against the Hungarian list's 14.85, so English takes one more
word. The test computes both sides from the embedded lists rather than comparing a constant with
itself, so shrinking a list or lowering a count fails.

**A third secret is NOT minted here and must not be added.** The customer **recovery code** is minted
by `felhom-agent` (`internal/escrow`) from the same EFF list, ten words, ≈129 bits — it has been
English since it was written. R-597's row listed it here; that was wrong. Writing a row for it in
this table would create a second definition of a secret the hub does not own, which is the drift
`backupTargetAbsentText` already demonstrates across two repos.

**Which language, and when.** The setup code follows `Store.CustomerLanguage` — the same order the
mail carrying it follows (reported → created-with → `hu`), so a code and its e-mail can never
disagree. Two consequences, both deliberate:

- **A household created as `hu` whose box later reports `en`** keeps every code already issued
  exactly as it was; the **next** code issued is English. A code is a hash on the box, never
  retranslated.
- **At customer creation there is no stored customer yet**, so the Owner passphrase generated on that
  form reads the language from **the form field**, not from `CustomerLanguage` — which would answer
  Hungarian for every English household created. The store's `createdLanguage` applies the same
  default to an absent value, so the two agree.

**The box needs no change for any of this.** It stores and compares a bcrypt hash of whatever was
minted and has no notion of which list the words came from — pinned by the controller's
`TestClaimAcceptsAnEnglishWordCode`. The TTL, the single-use generation and the five-attempt lockout
are untouched and language-blind.

**Count wording.** No claim mail states a word count; they say `Setup code: %s`. The only place a
count appeared was the bind page's passphrase hint, and its English half is now count-free ("The word
phrase you received from your operator during setup") — because "five words" stops being true for an
English household, and was already wrong for one whose passphrase predates this release. The
Hungarian „öt szó" is correct and unchanged.