# Felhom Hub — Changelog

## v0.68.0 — a credential re-issue finally re-arms the box (R-39 fleet fix) + wrapper drift is visible (R-50b(a)) (2026-07-21)

**Coupling, stated honestly: this release is SAFE for agents at 0.90.0** — the new descriptor field
is an unknown JSON key to them; they drop it and behave exactly as today (inert, not breaking).
**The re-arm and auth-honesty guarantees require agent >= 0.91.0.** Raise MinAgent to 0.91.0 only
after the fleet's agents have self-updated.

### The defect (R-39, fleet half)

An ep0 credential re-issue re-keys the **secret of an existing token**. `token_id`, `fingerprint`,
`datastore` and `namespace` all come back byte-identical — only the side-table `host_pbs_secrets`
row rotates. The agent's re-apply trigger is a change in the **descriptor content hash**
(`felhom-agent internal/pbsdr/manager.go` `descriptorHash`). Same hash → the converged agent
short-circuits → the fresh secret is never consumed → the box keeps presenting a revoked credential
→ **401 forever, while both tiers report `applied`**. Proven on the N100 2026-07-18: the agent's
`consumed-failed.json` hash was byte-identical to the `marker.json` written two minutes before the
re-issue.

### The fix

- **`host_pbs_secrets.generation`** — a monotonic per-host counter advanced by every fresh **mint**
  and by nothing else, stamped into the descriptor as `secret_generation`. That is now the only
  field a re-key moves, and it is what re-arms the agent.
  - **A re-stage deliberately does not advance it**: it re-arms the *same* secret, the descriptor
    content genuinely has not changed, and a bump would cause a pointless agent refetch loop.
  - `omitempty` is load-bearing — emitting a zero into every pre-existing descriptor would itself be
    a fleet-wide spurious re-apply.
  - **Deviation from the spec, deliberate:** the brief said to reuse "the new row's id … no schema
    change". There is no row id — the table is keyed by `host_id` and UPSERTed last-write-wins, so a
    new row never exists, and `created_at` collides for two mints in one second. An additive counter
    column (existing idempotent `ALTER TABLE` idiom) is the only monotonic source available.
- **`pbsdrheal` gains an `auth_failed` trigger** — a NEW trigger in the existing machine, not a new
  machine. A box whose credential PBS rejects is escalated to a fresh mint (never a re-stage: that
  re-feeds the secret PBS just rejected), through the **existing damping** — a 401 flap must not
  become a secret-minting chain. This closes the loop end to end: agent proves the 401 → hub re-keys
  → generation advances → descriptor hash moves → agent re-consumes.
- **`consumed_at` honesty gauge** — a staged secret left **unconsumed** past a 15-minute grace while
  the box reports `applied` is surfaced loudly with its own event. This is the exact July-18
  fingerprint and a disagreement **no single tier can detect alone**. Deliberately a *surface*, not
  a heal: auto-re-issuing here would mint a second secret on top of an unconsumed one — the
  mint/consume race R-39(a) already recorded.
- **Corrected a comment that stated a falsehood**: `ReissuePBSDR` claimed it refreshed the descriptor
  "with the NEW token_id/fingerprint". That is false for a re-key, and believing it is why nobody
  expected the descriptor to come back identical.

### R-50b(a) — wrapper drift is answerable

`ArtifactManifest.WrapperSHA256` + an operator field. Unlike the agent binary and the golden, the
PBS-DR wrapper is installed from `raw/branch/main` — unversioned, unpinned, absent from every
manifest — yet it is root-owned 0755 and the pinned sudoers vector. Agents (>= 0.91.0) report the
installed file's hash and the host page surfaces a mismatch. **An unknown on either side reads as
quiet, never as drift** — lighting every host amber on rollout day is how a warning becomes noise.
This does not fix the delivery channel; that stays R-50b(b)/(c).

### Tests

Store-level generation monotonicity, per-host isolation and restage-leaves-it-alone; descriptor
byte-change, `omitempty` and sibling-key round-trip; a **flow-level** test driving `ReissuePBSDR`
against a fake that models a real re-key; `auth_failed` escalate/debounce/recovery; the honesty
gauge incl. its grace window, the restage edge, the consumed case and the honest-stuck case; wrapper
drift incl. both unknown directions. **Two red-proofs run at the assertion level** (not the
compiler): removing the generation stamp makes the flow test fail with both byte-identical blocks
printed; removing the `auth_failed` arm makes the escalation tests fail with `reissues=0`.

## v0.67.0 — the hub stops keeping things to itself: auto-minted self-bind link, post-RESET staleness, unprovisioned-offsite warning (2026-07-18)

Four small items, each one a case where the hub already knew something and said nothing. Green:
`go build ./... && go vet ./... && go test ./...` all pass.

- **(a) The self-bind link is minted automatically — at customer creation AND at RESET completion**
  (R-36 sub-item). The box's console banner tells the customer to open „az e-mailben kapott link";
  until now that email existed only once the operator remembered to press *Send self-bind link*, so
  the banner could be instructing someone to look for something that did not exist. During the
  2026-07-18 rehearsal the box sat in pairing mode for ~11.7 minutes waiting on exactly that.
  `handleSelfBindLinkSend`'s body was extracted into a shared `mintAndSendSelfBindLink` core so the
  button and the two auto-mint call sites **cannot drift apart on the honesty rules**: F1 (no
  registered address → mint nothing, because a link nobody can receive is worse than none) and F2
  (send failed → delete the token, never leave it silently live). The auto-mint wrapper **never
  fails the operation it rides on** — a customer create that provisioned Cloudflare, offsite and PBS
  must not 500 because a courtesy email bounced; every outcome is logged instead, and the operator
  can still re-send from the Setup tab.
  **Gap found and closed while wiring this:** `PurgeCustomerResetDBState` does **not** clear
  `selfbind_tokens`, so a capability link minted *before* a RESET would have stayed live across it.
  A successful mint already replaces it (minting is delete-then-insert, single-active), but the skip
  paths would not have — so the wrapper now clears stale tokens on those paths too. The invariant is
  now: after auto-mint runs, the only live link is one it just issued, or none.

- **(b) Post-RESET staleness banner** (R-37). When a RESET **completed** after the newest report,
  every health figure on the customer page describes a lifecycle that no longer exists — and the
  page went on showing pre-RESET warnings as if they were current. It now says so, quoting the
  customer-facing phrasing („RESET óta nincs adat") and the reset's timestamp. Deliberately narrow:
  an in-flight reset does **not** trigger it (only a completed one), and it clears itself the moment
  a report arrives. Ties resolve to *stale* — SQLite timestamps are second-resolution, and a
  same-second report almost certainly arrived just before the reset destroyed what it describes;
  erring the other way would hide the banner exactly when it matters most.

- **(c) Unprovisioned-offsite warning** (R-36 interim). `offsite.enabled == true` with no
  descriptor (`type == ""`) is a real, stable, silent state: provisioning is *Save*-triggered
  (`applyOffsite`), and the re-enroll auto-re-issue deliberately skips an unprovisioned target, so
  nothing self-heals it. The page now names the state and the fix (press Save once, then verify),
  reusing the exact predicate the offsite re-issue handler already refuses on.

- **(d) `pbsdr_reissued` rendered an EMPTY flash box.** The flash key had no branch in the template,
  so re-issuing PBS credentials showed the operator a success box containing nothing — observed live
  on 2026-07-18. It now describes what was staged **and** carries the R-39 caveat: confirm
  `pvesm status` shows the entry *active*, because a converged agent can report `applied` while the
  storage still authenticates 401.

- **Styling:** new `.flash-warn` (amber, `--warn`/`--warn-dim` tokens) for the deviation tier between
  success and error — per the exception-color principle it appears ONLY on deviation, never on a
  healthy page.

- **Tests** (`customer_state_banners_test.go`, `selfbind_automint_test.go`) assert each banner is
  **absent** in the nominal cases as well as present in the deviating one — a banner that renders
  unconditionally is worse than none, because operators stop reading it. Auto-mint covers F1, F2,
  the no-mailer case, and the pre-RESET-token invariant. **Both red-proofed:** deleting the
  `pbsdr_reissued` branch reproduces the original empty-box bug, and neutering the staleness
  predicate fails the banner assertion. New store accessor `CountSelfBindTokens` (read-only, keyed
  by a customer id the operator already knows) makes the single-active invariant assertable.

**Not in this train:** the R-39 hub-side generation-bump fix the pre-travel task made conditional.
Its condition was **refuted** — `SetHostDesired` bumps unconditionally and `applyPBSDR` is
idempotent as documented; the real mechanism is the agent's descriptor-hash convergence, which needs
its own spec. Nothing was improvised here.


## v0.66.0 — Customer self-bind (R-27 slice 1): tokenized capability link + public two-factor `/bind/` page (2026-07-17)

Lets a customer bind their OWN freshly-installed appliance without the operator. Until now every
box booted from the universal secret-free ISO (R-21 slice C) had to be bound by the operator on the
Hosts page; this adds the self-service path. **Viktor's three rulings, each honoured verbatim:**
(a) *"only their own visible"* → the customer proves possession with the **console pairing code**
shown on the box screen — **no appliance list is ever rendered** on any public surface; (b) *first-box
entry* → an **operator-sent 7-day tokenized capability link** over Hungarian email (the claim-engine
delivery pattern, a sibling sender — NOT routed through the claim engine); (c) **lockout after 5
failed attempts** → the token locks and the page says *"call support"*. Wrong code and wrong
passphrase produce **one identical generic failure** (no oracle); an expired link falls back to
operator-bind, unchanged. Does **not** touch the controller or agent. Green: `go build/vet/test`
(full hub suite + 9 new self-bind tests), all 4 red-proofs verified red-then-green, hub confirm gate.

- **Pairing code (Part 1).** `POST /api/v1/appliance/register` now returns an additive `pairing_code`
  (6 chars, ambiguity-free alphabet, `ABC-234` display) minted once at first registration and stable
  across the idempotent re-register/upsert (backfilled if a pre-existing row had none). Persisted on
  `appliance_registrations.pairing_code`; shown in the operator Hosts "Unclaimed appliances" table.
  The bootstrap (`felhom-bootstrap.sh`, ISO v1.20.0) parses it and prints a Hungarian **console
  banner** to `/dev/console` so the customer can read it off the screen. Old ISOs ignore the field;
  an old hub omits it and the banner prints nothing — additive both ways.
- **Capability token (Part 2).** New `selfbind_tokens` table: `sha256(token)` at rest (never the
  token), **single-active per customer** (a re-mint deletes the prior row in one tx — an old link dies
  the instant a new one is sent), `attempts` locking at 5, one-shot `consumed_at`, `emailed_at`
  honesty. Operator **"Send self-bind link"** button on the customer Setup tab
  (`POST /customers/{id}/selfbind-link`) mints a 256-bit token and emails
  `https://hub.felhom.eu/bind/<token>` (Hungarian, adult tone, no emoji, names both factors + the
  7-day + 5-attempt limits). **F1** (no registered email → nothing minted, LOUD flash) and **F2**
  (email send fails → the just-minted token is deleted, not left silently live) are both honest.
- **Public bind page (Part 3) — THE TRAP (§9.2).** One new public prefix `/bind/`, exempted from
  operator auth AND CSRF at the two gate sites the `/login` exemption occupies, via a **single**
  predicate `isPublicBindPath` (matched tightly: trailing slash → no sibling like `/bindsecret`; the
  ServeMux `..`-cleans before we see the path → no traversal reach; the handler also rejects a token
  containing `/`). `GET` renders form/consumed/locked/expired; `POST` normalizes both inputs, compares
  **both factors unconditionally** (constant-time passphrase vs the customer's retrieval passphrase;
  the ONE bindable appliance carrying the console code), then decides — identical generic failure
  either way. On success: the **same `BindAppliance`** the operator uses, a provenance event with
  source `customer_selfbind`, one-shot consume, and *"A doboz kb. egy percen belül folytatja a
  telepítést."* The box's ~30 s appliance poll picks up the delivery. Own per-IP rate limiter; the
  page is self-contained (it cannot link `/style.css`, which is itself operator-gated). No hub
  customer-login/session was built — the URL capability token IS the auth model; a cross-site POST
  without both secrets only burns attempts (accepted + documented).
- **Tests + red-proofs (Part 4).** Scenarios A–F + F1/F2 (9 tests). The **4 red-proofs** were each
  applied and confirmed to turn exactly their scenario red, then reverted green: lockout removed → C1;
  oracle introduced → B; `/bind/` prefix widened (drop the slash) → E (and D); single-active DELETE
  dropped → C4. The passphrase is never logged/echoed/persisted; only attempt COUNTS are logged
  (`self-bind attempt N/5 … token <8hex>…`); the raw link token never enters logs or events.
- **GC verdict (spec §3):** there is **no appliance-staleness GC** in the hub (`applianceStaleAfter`
  is a DISPLAY badge only; `pruneAll`/`PurgeExpiredLogBundles` touch reports/log-bundles, not
  appliances or selfbind tokens). The 7-day token TTL therefore stands alone and needs no reaper —
  single-active-per-customer means at most one row per customer, superseded rows are deleted on
  re-mint, and an expired row simply reads as expired (no security or storage pressure).

## v0.65.0 — PBS DR storage visibility (ep0 `usage` op) + Offsite tab split (Restic / PBS DR) + dual dashboard gauges (R-5) (2026-07-17)

Makes the **PBS DR** storage visible like the restic pool box already is (v0.64.0), the two clearly
differentiated. The scoping correction: "the restic box" and "the PBS box" are NOT two Hetzner Storage
Boxes — **restic** = subaccounts on the shared Hetzner Storage Box (Hetzner API, v0.64.0); **PBS DR** = a
PBS datastore (`felhom-offsite`) on the ep0 endpoint VM (NO Hetzner API). This adds the hub's read of the
PBS datastore fill via **Option A (Viktor-ruled):** a new read-only `usage` op on the `felhom-tenantsync`
ep0 forced command (the structural twin of the existing `fingerprint` op), polled by a new hub checker on
the same 15-min throttle; splits the Offsite page into **Restic / PBS DR tabs**; and puts **two dashboard
gauges** (restic %, PBS %). READ-ONLY against ep0 and Hetzner. Green: `go build/vet/test` + `bash -n` +
the script harness; hub confirm gate OK.

- **Phase-0 probe (gate, PASSED):** on ep0 (PBS 4.2.3), `df -B1 --output=size,used,avail <datastore path>`
  (path from `proxmox-backup-manager datastore list --output-format json`) yields the datastore
  total/used/avail in **bytes** (live: 39990112256 / 7627939840 / 30686326784 → ~19%), read-only, in the
  existing sudo context, no admin token. (PBS 4.2 has no native `datastore usage` command.)
- **`scripts/felhom-tenantsync.sh` → v1.2.0:** a read-only `usage` short-circuit (before the admin-token
  generation, like `fingerprint`) → `{"status":"ok","total","used","avail"}`. **No customer_id, no admin
  token, NO mutation.**
- **`tenantsync.Client.Usage()`** (`internal/tenantsync/client.go`): `BoxUsage{Total,Used,Avail}` + the
  op; an endpoint ≤ v1.1.0 answers `bad_request "unknown op"` → typed `ErrUsageUnsupported` (the
  graceful-degradation signal).
- **`monitor.PBSDRBoxChecker`** (`internal/monitor/pbsdr_box.go`, new): clones OffsiteBoxChecker over a
  `usageReader` seam (the tenantsync client) — 15-min throttle, cached `PBSBoxSnapshot`, escalation-only
  `pbsdr_box_fill` on the customer-less `"pbsdr-box"` scope (operator channel only, no SaveEvent),
  recovery re-arm. FILL ONLY (PBS uses namespaces, not quotas — no oversubscription). **THREE states:**
  `ok` (bands drive), `unavailable` (ErrUsageUnsupported — expected pre-update, neutral, NO alert, logged
  once, the gauge shows n/a), `degraded` (exec failed — keep last snapshot, no band transition).
- **Config + wiring:** `Alerting.PBSDRBoxFill{Warn,Crit}Percent` (default 80/90, independently tunable);
  the checker is built ONLY when the tenantsync client exists (shares it), registered in the 60 s sweep,
  snapshot handed to the web server. **Graceful degradation: the hub deploy is INDEPENDENT of the ep0
  update** — a hub v0.65.0 against an ep0 still on v1.1.0 shows the honest "n/a", lighting up on the next
  poll once ep0 is updated (no hub redeploy).
- **Web (`internal/web/pbsdr_box.go` new, `offsite.go`, `templates/offsite.html`, `dashboard.html`,
  `style.css`):** the Offsite page splits into **Restic** (the v0.64.0 pool-box panel + per-customer rows)
  and **PBS DR** (a new datastore panel — capacity/used/fill bar with band + the endpoint cards, which
  belong here: the endpoint IS the PBS host) hash tabs (server-rendered, no JS dependency for the data).
  The single dashboard tile becomes two gauges — **RESTIC** (pct·ratio) and **PBS DR** (pct; "n/a" when
  unavailable) — each band-colored, each linking to its tab.
- **Tests + red-proofs:** 10 Go tests (Usage parse + unknown-op→typed; checker throttle/bands/pbsdr-box
  operator-only/unavailable-no-alert/degraded-keeps-last; PBS panel render × ok/unavailable/not-configured)
  + a bash script harness (usage JSON + exit 0 + **zero mutation** + provision regression). Red-proofs
  (run-fail-restore): the op emitting a mutation → the harness zero-mutation assertion fails; the
  escalation-only guard removed → in-band re-emit fails; unavailable driving a band → the no-alert test
  fails. All confirmed red, then restored.

## v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5) (2026-07-17)

Ships **R-5**: the operator sees the shared pool box's real state on the hub — **total box fill vs
capacity**, **Σ(shared soft quotas) vs capacity** (the oversubscription ratio), **per-customer usage/quota
bars**, and a **box-level operator alert** (fill % + oversubscription ratio) riding the existing
dispatcher's operator channel. Per-customer fill alerts already existed (OffsiteChecker, 90/95% of each
quota); the box-level aggregate was the gap — the operator's early warning that the *pool itself* is
filling, before any single customer breaches. All READ-ONLY against Hetzner (GET only). Green:
`go build/vet/test` all pass; hub confirm gate OK.

- **Phase-0 probe (gate, PASSED):** one authenticated GET of the live pool box (611714) pinned the API
  shape — capacity is `storage_box_type.size` (1 TiB for bx11), usage is a `stats` object
  (`size`/`size_data`/`size_snapshots`), all bytes; our token reads it (200). The type extension mirrors it.
- **`hetznerapi` (`hetznerapi.go`, `fake.go`):** additive `StorageBoxType` + `StorageBoxStats` sub-structs
  on `StorageBox` (no existing field/method changed); fake carries them + a `GetBoxCalls` counter. Golden
  decode test against the (redacted) probe capture.
- **`monitor.OffsiteBoxChecker` (`offsite_box.go`, new):** the OffsiteChecker-sibling for the box as a
  whole — **fetch-throttled** (one Hetzner GET per 15 min; ≈4/hour, never per-sweep or per-page-load),
  cached `BoxSnapshot`, escalation-only emits with silent recovery re-arm. Two independent signals: FILL
  (used/capacity, warn 80% / crit 90%) and OVERSUBSCRIPTION (Σ shared+enabled quotas / capacity, warn
  2.0×) — both can fire, neither masks the other. Σ(quota) is read from the authoritative ConfigJSON
  `Descriptor` (`offsite.ReadDescriptor`, new), NEVER the report echo; dedicated + disabled customers
  excluded. Events carry the customer-less scope `"pool-box"` → operator channel ONLY (`processCustomer`
  no-ops on it) and are NOT SaveEvent'd (no customer row to key them). A failed fetch keeps the last
  snapshot marked degraded — missing data never becomes 0% and never drives a band transition.
- **Config (`cmd/hub/main.go`):** `Alerting.OffsiteBoxFillWarnPercent` (80) / `OffsiteBoxFillCritPercent`
  (90) / `OffsiteOversubWarnRatio` (2.0), plumbed like `StorageFill*`. The checker is constructed inside
  the existing `HETZNER_TOKEN` branch (shares the client), registered in the 60 s sweep, and its snapshot
  handed to the web server. **Thresholds are Claude's encoding of the starter suggestion — Viktor's ruling
  pending; the named keys are the one-line flip.**
- **Web surfaces (`web/offsite_box.go` new, `templates/offsite.html`, `dashboard.html`, `style.css`):** an
  Offsite-tab panel (capacity, used with data/snapshot split, fill bar, Σ quotas + ratio, fetched-at, +
  per-customer rows sorted by usage — shared with a usage/quota bar, dedicated listed without one, no-report
  customers show "no usage reported yet") and a compact Dashboard tile (`fill% · ratio`, band-colored,
  linking to /offsite). The web layer reads only the cached snapshot — it NEVER fetches. Nil provider → both
  render an honest "not configured". Exception color (neutral/amber/red, no green).
- **Tests + red-proofs (run-fail-restore):** 10 new tests (throttle ≈4-not-≈60, Σ/ratio truth table,
  band transitions incl. in-band no-re-emit + recovery re-arm + oversub independence + pool-box operator-only
  scope, failed-fetch honesty, zero-capacity guard, golden decode, panel render × "not configured"/"with
  data"/"no usage reported yet"). Red-proofs: (i) drop the throttle → ≈60 calls; (ii) sum dedicated/disabled
  → wrong Σ; (iii) drop the escalation-only guard → in-band re-emit; (iv) zero the snapshot on a failed fetch
  → lost last-known. All confirmed red, then restored.

## v0.63.0 — system-initiated immediacy: wire the proven poke/bump notifiers into every mutation site that lacked one (2026-07-17)

The immediate-sync arc (Dir-1 trigger, Dir-2b wait channel, Dir-2a agent poke) covered only
**operator-initiated** desired-state changes. **System-initiated** mutations still bumped the
generation silently, so a freshly onboarded box waited a full agent tick (≤15 min) for state the hub
had already minted — observed live during slice-C onboarding. This wires the existing, live-proven
notifiers (`poke.Notifier` for the agent plane, `intent.Hub.Bump` for the controller plane) into
every system-initiated site that lacked one. No new mechanism — call-site wiring only.

- **Agent-plane pokes (`internal/web/pbsdr.go`):** `PBSDRAutoProvision` (the exact observed lag — the
  WG-registration hands-free provision) now pokes on success; `ReissuePBSDR` (the shared core, which
  also gives the **pbsdrheal reconciler's escalation** its immediacy with zero reconciler changes) and
  the operator button `handlePBSDRReissue` poke after their descriptor bump. All fire ONLY after the
  successful `SetHostDesired`, never on a blocked/error path.
- **Agent-plane Poker seam (`internal/api/handler.go`, `internal/api/wg.go`):** a new nil-safe `Poker`
  interface (`PokeHost`/`PokeAllHosts`, satisfied by `*poke.Notifier`) + `SetPoker`. `handleAdminSetDesiredState`
  pokes the target host after a successful admin desired-state write; `handleAdminSetOperatorPeer` fires
  a fleet `PokeAllHosts` — but only when `BumpAllHostGenerations` succeeded (fire-after-commit).
- **Controller-plane bump (`internal/api/handler.go`):** `reissueOnReenroll` (the clean-slate F2 claim +
  F3 offsite re-issue) now `intentHub.Bump`s the customer so a long-polling controller wakes in seconds
  instead of on the 15-min cycle. Nil-guarded; one unconditional bump (coalesced, over-bump harmless).
- **main.go:** one `poke.Notifier` instance now feeds BOTH planes' system sites — `webServer.SetPoke(n)`
  **and** `apiHandler.SetPoker(n)`; startup log: "web + api admin seams armed".
- **Deliberate non-sites (unchanged, documented in REPORT audit table):** WG peer *register* (box tunnel
  doesn't exist pre-fetch — a poke is undeliverable; the agent fast-tick SECONDARY owns this leg), WG peer
  *delete* (the mutation removes the transport), and the **pbsdrheal Restage** path (no generation bump →
  the agent's 60 s pbsdr ticker is the pickup path; a poke there is a verified no-op). `internal/pbsdrheal/`
  is byte-unchanged.
- **Tests + red-proofs (run-fail-revert):** 10 new non-hollow tests. Web (async, channel-synchronized
  fake sender): auto-provision pokes the resolved WG /32, blocked-precondition pokes nothing, reissue-core
  + operator-button poke, a failed reissue pokes nothing. API (synchronous fake Poker): admin-set pokes
  the target host only (0 on invalid-JSON), operator-peer fires one fleet poke, nil-seam no-panic; re-enroll
  advances the intent generation, nil-hub no-panic. Red-proofs demonstrated one representative removal per
  group (A/B web pokes, C admin poke, D re-enroll bump) — each FAILED red, then restored. Green:
  `go build/vet/test` all pass.

## v0.62.0 — R-21 slice C: the universal ISO — unclaimed-appliance registration + operator bind + one-shot delivery (2026-07-17)

The hub half of the universal, **secret-free** bare-metal ISO. A box booted from the generic ISO
registers itself as an UNCLAIMED APPLIANCE; the operator binds it to a customer on the Hosts page; the
hub delivers the customer-id + retrieval passphrase on the box's next poll, ONCE. The distributed ISO
carries no customer secret (§4.4).

- **Store (`internal/store/appliance.go`, new):** `appliance_registrations` keyed by **(uuid, mac_set)**
  — serials are unusable (N100 DMI "Default string") and cheap boards duplicate SMBIOS UUIDs, so the
  MAC set is the tiebreaker (same uuid + different mac-set = distinct appliance). `token_hash` = sha256
  of the appliance token (the token itself is never stored). `RegisterAppliance` (idempotent upsert;
  sticky-discard), `ApplianceByToken`, `BindAppliance`, `MarkApplianceDelivered` (atomic one-shot
  bound→delivered), `DiscardAppliance` (invalidates the token), `ListUnclaimedAppliances`. The table's
  own timestamps ARE the pre-bind provenance (no customer to scope an events row to yet).
- **API (`internal/api/appliance.go`, new):** `POST /api/v1/appliance/register` — the ONE
  unauthenticated endpoint, per-IP rate-limited, returns a random 256-bit appliance token. `GET
  /api/v1/appliance/poll` (Bearer token): unknown/discarded → **404** (no oracle), unbound → **204**,
  bound → **200** + credentials (consumed once), delivered → **410**. The passphrase is read live from
  `customer_configs` (plaintext, as the day-0 command already needs it) and never logged.
- **Web (`internal/web/appliances.go`, new):** the Hosts page grows an "Unclaimed appliances" section
  (uuid, MACs, hw, **SSH host-key fingerprints**, first/last seen, stale >7d badge) with **BIND**
  (customer picker showing host counts — display only, never a gate) and **DISCARD**. Bind stages the
  delivery + emits `appliance_bound`; delivery emits `appliance_credential_delivered`.
- **Red-proofs (run-fail-revert):** the one-shot delivery (defeat the bound→delivered flip → second
  poll re-delivers the passphrase → FAIL) and register idempotency (drop the upsert → duplicate/UNIQUE
  violation → FAIL), both proven red then restored; plus 404-no-oracle + sticky-discard, bind
  staging/refusal, and the render test. Green: `go build/vet/test`; hub confirm gate OK.

## v0.61.0 — Customer RESET: the middle lifecycle tier (2026-07-17)

One operator action returns a customer to **pre-first-install**: every OPERATIONAL trace dies (offsite
repo, PBS namespace + backups, DR recipe, one-time secret, claim state, retained escrow custody), while
**identity and the basic config survive** (the `customer_configs` row, all provenance rows, and the
audit-event stream). It sits between the two existing tiers — *host delete* (< RESET) and *customer
Delete* (> RESET, the one true purge point). Viktor's rulings: (1) destroying retained escrow custody
gets its **own** separate acknowledgment; (2) RESET clears claim state (a fresh code next onboarding);
(3) RESET **refuses while any host row exists** (delete hosts first — reset never deletes hosts); (4)
the confirm surface shows a **live-counted** inventory.

Orchestration discipline (spec §3): external teardown FIRST, DB purge LAST (publish-last), every leg
idempotent → a partial run is simply re-run from the top; a failed external leg is a clean journal
entry and the DB purge (which erases the descriptors that say what still needs tearing down) is
withheld until every external leg is `ok`. Provenance + events are NEVER wiped.

- **Store** (`internal/store/customer_reset.go`, new): `customer_resets` journal table (per-attempt,
  per-leg status, resumable); `CustomerResetInventory` (live counts: hosts, retained blobs via the
  F-14 `host_deletions` UNION, dr_recipe/one-time-secret/claim presence); `Start/UpdateResetLeg/Finish/
  LatestCustomerReset`; `PurgeCustomerResetDBState` (ack-gated escrow-blob delete + one-time-secret,
  dr_recipe, log bundles — never touches identity/provenance/events); `DeleteClaim` primitive.
- **Claim** (`internal/claim/engine.go`): `ResetToUnclaimed` DELETES the claim row so `EnsureIssued`
  mints a fresh first code on the next onboarding (no parallel revoked-flag, no stale generation).
- **Offsite** (`internal/offsite/offsite.go`): `Deprovision` DELETES the labelled sub-account/box
  (idempotent — label-lookup, `len==0` = already gone); `OffsiteIdentifier` (preview name);
  `ClearProvisionedDescriptor` (keeps the tier CHOICE `enabled/type/quota/box_type`, drops every
  provisioned field). **PBS** (`internal/tenantsync/client.go` + `scripts/felhom-tenantsync.sh`
  `deprovision` op): destroys the customer's namespace + all backup groups + token; the shared
  `felhom@pbs` user is never touched; idempotent.
- **Web** (`internal/web/customer_reset.go`, new): `GET /configs/{id}/reset` → live inventory JSON;
  `POST` → the orchestration (precondition + typed-id + escrow-ack gates BEFORE any write/external
  call). A distinct **amber** RESET card on the customer page (separate from the red Danger-zone
  Delete), with the typed-id confirm + the separate escrow-custody ack row.
- **Red-proofs**: ack-gate (defeat → reset proceeds & destroys blobs → FAIL); partial-failure
  resumability (purge-not-withheld → DB purged despite external failure → FAIL); both proven red then
  restored. Plus store ack-gating, journal round-trip, offsite Deprovision idempotency + descriptor
  clear, and the RESET-card render test. Green: `go build ./... && go vet ./... && go test ./...`.

## v0.60.1 — host deletion DEMOTES escrow custody (never destroys) + S6b obsolete (2026-07-17)

Closes the deletion-path gap in v0.60.0's review: `DeleteHost(deleteEscrow=true)` was still DELETING
escrow rows (the same-customer reinstall flow funnels the operator straight into that tick).
Principle (Viktor's standing ruling): host deletion is a lifecycle event — blob custody survives it;
the customer Danger-zone Delete is the one true purge point. Green:
`go build ./... && go vet ./... && go test ./...`.

- **Scenario A — host delete demotes, never destroys.** `DeleteHost(deleteEscrow=true)` now DEMOTES
  the current `host_escrow` row into `host_escrow_superseded` (copy-BEFORE-delete, same tx) and SPARES
  existing superseded rows — no operator path through host lifecycle can lose a blob. Reuses THE one
  escrow row-copy routine (`demoteCurrentEscrowTx`, also used by `SaveHostEscrow`). The F-14
  provenance row + gate semantics are unchanged (wording updated: demotion, not destruction). Edge:
  no escrow row → unchanged; the `ErrHostEscrowPresent` refusal without the flag is unchanged.
  Red-proof `TestDeleteHost_DemotesEscrowNeverDestroys`.
- **Scenario B — customer delete is the purge point.** `DeleteCustomerConfig` (which previously deleted
  ONLY the `customer_configs` row) now, in one tx, purges `host_escrow` AND `host_escrow_superseded`
  for all the customer's hosts — INCLUDING already-deleted hosts (resolved via the F-14
  `host_deletions` provenance) so a host-delete-then-customer-delete ordering leaves nothing orphaned.
  Danger-zone copy states it. Red-proof `TestDeleteCustomer_PurgesEscrowCustody`.
- **Scenario C — wording.** The host-delete escrow checkbox now reads "Move key escrow to retained
  custody (required when escrow present)…"; the refusal message + customer Danger-zone copy match.
  Guard test `TestHostDeleteEscrowLabel_DemotionWording`.
- **Scenario D — S6b verdict (docs): OBSOLETE.** Re-enrolling an existing host_id upserts cleanly
  (`UpsertHost` ON CONFLICT DO UPDATE, `store.go`; `handleAdminCreateHost` has no duplicate refusal) +
  the v0.57.0 re-enroll arc auto-fires the re-issues → no manual stale-host deletion needed before
  re-enroll. Scenario A also makes the funnel harmless either way. ROADMAP R-3 refined.
- **Scope:** hub-only; no controller/agent change; ACK assembly + upload supersede path untouched.
  Deploy: bump `manifests/hub.yaml` tag to `0.60.1` and sync.

## v0.60.0 — offsite continuity Part B: superseded-escrow retention (data-first) (2026-07-17)

Closes the data-loss half of the reinstall-orphaned-repo incident: `SaveHostEscrow`'s destructive
`ON CONFLICT` overwrite meant a new escrow blob DESTROYED the old passphrase's only copy — so 18
snapshots keyed under the old password became unrecoverable. Viktor's ruling (data protection first):
**retain superseded blobs** so the old passphrase stays customer-R-recoverable. Pairs with controller
v0.142.0 (Part A orphaned-repo guard). Green: `go build ./... && go vet ./... && go test ./...`.

- **`host_escrow_superseded` (new table) + `SaveHostEscrow` rewrite.** On upload, if the current row
  seals a DIFFERENT `restic_pw_sha256`, the old row is COPIED into the history table (in one tx)
  BEFORE the current row is overwritten; a same-sha re-upload (idempotent re-ceremony) refreshes the
  current row and creates NO supersede row. `SaveHostEscrow` now returns `superseded bool`. Retain
  ALL (no pruning — the blobs are tiny + R-encrypted, custody unchanged); the hub still never
  decrypts. **ACK/restore-serving read the CURRENT row (`GetHostEscrow`) — unchanged.** New
  `CountSupersededEscrow` / `ListSupersededEscrow` (the latter seeds the future R-26 recovery flow).
  `DeleteHost(deleteEscrow=true)` also drops the retained rows.
- **Surfaces.** Upload handler emits the hub-internal `escrow_superseded` audit event + logs the
  retained count; the operator host-detail DR/Backup panel shows "N superseded escrow blob(s)
  retained". Registered `offbox_repo_orphaned` / `offbox_repo_reset` (controller v0.142.0 pushes) in
  `allowedEventTypes` + `customerMessages`.
- Red-proof `TestSaveHostEscrow_RetainsSuperseded` (pre-fix destructive overwrite → old blob gone →
  FAIL; fixed → retained + retrievable; same-sha idempotent).
- **Deploy:** bump `manifests/hub.yaml` image tag to `0.60.0` and sync.

## v0.59.0 — Direction-2a: agent-plane immediate-sync poke sender + ep0 felhom-poke surface (2026-07-16)

Implements the AGENT-plane half of `documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`
option (a): an operator agent-plane change (a pbsdr descriptor, a MinAgent floor) now nudges the box
in **seconds** via a CONTENTLESS UDP poke relayed hub → ep0 forced-command → wg0-origin → the agent's
poke listener (felhom-agent v0.89.0). The spike reserved this transport for the agent plane and
measured it at ~0.42 s/poke. Complements v0.58.0's `internal/intent` long-poll wait channel
(Direction-2b, the CONTROLLER plane): intent is customer-keyed and wakes the config puller; poke is
host-keyed and nudges the agent. Both are fire-and-forget; the 15-min report cycle stays the
guarantee. Pairs with felhom-agent v0.89.0 (the listener).

- **`internal/poke` — the SSH poke sender + notifier.** Third structural sibling of
  `internal/wgsync`/`internal/tenantsync`: a pinned-host-key (`ssh.FixedHostKey`, algorithm-pinned)
  in-process SSH client that reuses the peersync endpoint + host key, its OWN forced-command key.
  `Client.Poke(ctx, boxWGIP)` refuses any target outside `10.77.0.0/24` BEFORE dialing, then SSHes to
  ep0's `felhom-poke` with the box's WG /32 as the command string (→ `$SSH_ORIGINAL_COMMAND`); the
  forced command sends one empty UDP datagram to `<ip>:51822`. `Notifier.PokeHost`/`PokeAllHosts`
  are fire-and-forget (detached goroutine, nil-receiver-safe) — a poke NEVER blocks or fails the
  operator save; a missing peer / SSH error is logged and the report cycle reconciles. Tests:
  resolved-IP send, no-peer/store-error no-send, nil no-op, `Poke` pre-dial non-WG refusal.
- **Wiring** (`internal/web/server.go`, `internal/web/pbsdr.go`, `internal/web/configs.go`,
  `cmd/hub/main.go`): `Server.SetPoke`; `applyPBSDR` fires `PokeHost(host.HostID)` after each
  generation-bumping descriptor save (disable, storage-id/re-enable change, fresh provision);
  `handleSetArtifacts` (the MinAgent-floor / vouched-agent save) fires `PokeAllHosts()`. **Source note
  (spec landmark vs source):** the artifact-manifest save does NOT itself bump per-host desired
  generation — the agent self-update dispatches via signed-ops on the next report — so the fleet poke
  there accelerates the next report cycle where the floor is applied, rather than delivering a
  desired-state delta. The sender is env-configured (`POKE_SSH_KEY_FILE`, reusing
  `WG_ENDPOINT_SSH_ADDR`/`_HOSTKEY`/user); absent Secret → "agent-plane poke disabled".
- **ep0 surface** (`scripts/felhom-poke.sh` v1.0.0 + `documentation/runbooks/offsite-endpoint.md`
  §11): a NON-root (`felhom-peersync`, no sudoers grant — a datagram needs no privilege)
  forced-command that validates `$SSH_ORIGINAL_COMMAND` to the WG /24 and sends one empty datagram
  from wg0. Contentless + confined (the WG kernel independently refuses non-peer /32s — spike P1
  EKEYREJECTED). Port **51822** is a shared cross-repo constant.
- **`manifests/hub.yaml`:** `POKE_SSH_KEY_FILE` env + optional `Secret/agent-poke` mounted 0400 at
  `/etc/hub-secrets/agent-poke/key` (the private key stored out-of-band, per §11). Bump the image tag
  to `0.59.0` and sync.

## v0.58.0 — Direction-2 immediate-sync: the hub→box "sync now" wait channel (2026-07-16)

Implements option (b) of `documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`: an
operator action on the hub now reaches the box in **seconds** instead of on the next ~15-min report
cycle. The box holds a hanging authenticated `GET /api/v1/wait` over the existing outbound ingress;
the hub completes it the instant any operator intent lands for that customer. The box then fires its
ordinary out-of-cycle report — the ACK delivers config/escrow/claim/floor through the UNCHANGED
machinery. The 15-min cycle stays the reconciliation backbone; every wait failure degrades to it.
Pairs with controller v0.140.0 (the long-poll client). Ground truth #1 holds: the box pulls even the
wake-up; the hub never connects inbound and no state ever rides the wait response.

- **`internal/intent` — the in-memory operator-intent notifier.** A per-customer generation counter
  with a waiter registry: `Bump(customerID)` advances the generation and wakes every registered
  waiter (a burst coalesces into ONE completion carrying the LATEST generation — a counter, not a
  per-bump queue); `Wait(ctx, customerID, lastSeen, maxHold)` returns the instant the generation
  differs from `lastSeen`, on ctx-cancel, on `maxHold`, or on `Close`. A pre-register gen-check
  closes the bump-before-connect race (a bump is never lost). In-memory BY DESIGN — a hub restart
  resets generations; the box compares with `!=`, so a restart costs exactly one harmless full-state
  report, never a storm. No persistence, no store schema. Red-proofs: counter-vs-queue (return the
  as-of-register snapshot → `TestWait_CoalescesBurstToLatestGen` fails) and the race-closer (drop the
  pre-register check → `TestWait_RaceCloser_BumpBeforeWaitNotLost` fails); both run-fail-reverted.
- **`GET /api/v1/wait` (api/wait.go).** Authed via `checkAuthCustomer`; per-customer only (a global
  operator key → 400; the customer is resolved from the key, no `customer_id` parameter is accepted,
  so A can never observe B). Holds up to **240 s**, writing a **25 s heartbeat newline** while it
  waits. nginx's `proxy_read_timeout` is measured BETWEEN upstream reads, so the heartbeat keeps the
  default 60 s from ever firing — **no ingress annotation / manifest timeout change is needed** (the
  transport spike measured that ceiling; §13 proves the heartbeat defeats it live). The response is
  contentless — a single `{"gen":N}` line. The connection's write deadline is lifted per-request via
  `http.NewResponseController().SetWriteDeadline` (the global `http.Server.WriteTimeout` of 60 s is
  deliberately untouched).
- **Intent bumps (web).** Every operator-intent handler bumps the customer's generation AFTER its
  successful store write (fire-after-commit, never on an error path), via nil-safe `s.bumpIntent`:
  config create/update/delete, claim resend, offsite re-issue (UI + the re-enroll seam), offsite
  freeze/unfreeze, retrieval-password regen, block/unblock, per-customer floor, global floor (bumps
  every config-managed customer), controller log-tail request, and the CONTROLLER log-bundle request
  (the AGENT ring rides the heartbeat envelope — a separate plane, deliberately not bumped).
- **Wiring + shutdown.** One `intent.New()` in `main.go`, injected into both the web server and the
  API handler; `intentHub.Close()` runs before `server.Shutdown` so held waits complete instantly
  instead of eating the 15 s grace window. nil-safe throughout (an unset hub → wait 503, bumps no-op).

## v0.57.0 — reinstall-of-existing-customer arc: claim continuity, offsite re-issue, escrow honesty (2026-07-16)

Closes the N100 physical-run findings F2/F3 and the correctness edge behind F4→2.3
(`documentation/tests/VALIDATION-n100-baremetal-2026-07-16.md`). When an existing customer's box is
clean-slate reinstalled, hub and box previously disagreed about claim, offsite, and escrow state.
This makes the reinstall path first-class — the Peti (R-1) convergence prerequisite.

- **F2 — claim continuity.** `claim.Engine.ReissueForReenroll` rides the existing rotation semantics:
  for a CLAIMED customer whose box re-enrolls (fresh, passwordless), it bumps the generation ONCE and
  emails a RESET code (delivery via the existing report ACK) — the customer no longer has to hunt for
  the manual "request a new code" button. No-op for an unclaimed customer (first-provision path owns
  the code). Hooked at the host-enroll **mint path** (`handleHostEnroll`), which fires exactly once per
  fresh host record — the single-bump-per-re-enroll guarantee. Emits `claim_reissued_reenroll`.
  *(Fork verdict, source-verified: the hub stores only the claim code + a claimed boolean — never the
  password hash, which is controller-owned by the arc's design. So fork B, not A.)*
- **F3 — offsite continuity.** The re-enroll mint path also calls the same machinery as the manual
  "Re-issue offsite credentials" button (`web.Server.ReissueOffsiteForCustomer`, wired to the api
  handler via `SetOffsiteReissuer`) — the one-time offsite password only ever reached the OLD
  controller, so the fresh box gets a fresh one and a `ConfigVersion` bump. Emits `offsite_reissued`.
- **2.3 — escrow honesty (correctness; red-proofed).** Re-issuing offsite credentials changes the
  restic repo password, so any existing key-escrow blob is now STALE (a recovery code minted against
  it would decrypt a password that no longer opens the repo). `offsite.ReissueCredentials` now marks
  the escrow stale (`store.MarkEscrowStale`, cleared by the next ceremony via `SaveHostEscrow`); the
  ACK **withholds** the now-mismatched `restic_pw_sha256` so the controller cannot auto-confirm against
  a dead key, and the DR-tier checklist shows **stale** instead of "ceremony done." Emits `escrow_stale`.
  Red-proof: with `MarkEscrowStale` gutted, the hub keeps advertising ceremony-done after a re-issue →
  `TestReissue_InvalidatesEscrow` FAILS; restored → passes.
- **Out of scope (reported):** F4's general installer fix is NOT feasible — the DR storage id lives in
  the agent-domain pbs_dr descriptor (provisioned post-WG), not the installer-fetched config, and the
  underlying block is the agent's token-auth pre-check 403ing before its own root-run `felhom-pbs-apply
  grant`. Root fix is agent-side (ROADMAP agent-train item); the demo was unblocked live with a
  one-shot ACL grant. Controller (Part 3) unchanged: its escrow prereqs are already fetched live from
  the agent, so F4/Part-0 alone restore them (3.1 spec premise contradicted by source). scripts
  unchanged (v1.16.0).

## v0.56.0 — PBS-DR self-heal reconciler (re-stage a consumable secret) (2026-07-15)

Implements `SPIKE-pbsdr-selfheal-2026-07-15` (`e8f8c44`). **⚠️ ARCHITECTURE IMPACT:** before this,
there was **no automatic recovery** for the commonest real event — a customer box re-installed /
restored / rolled back onto its **stable `host_id`**. The hub kept the durable `pbs_dr` descriptor
(enabled) + a durable **consumed** one-time secret; the WG peer still existed (same pubkey →
`changed==false`, so the provision cascade could not re-fire) and `applyPBSDR`'s "already
provisioned → no-op" meant even a config re-save minted nothing. The agent sat in
`pbs_dr.state="waiting_secret"` forever — PBS-DR never converged, so escrow could not run and offsite
never armed. The spike proved (SQ-2b′) the **missing piece is a consumable secret, not the
descriptor**: re-staging the stored secret converged a stuck box in one ~30 s agent tick, using the
existing ep0 token, **zero churn**. This reconciler closes that gap.

- **`internal/pbsdrheal/reconciler.go` (new):** a hub periodic reconciler (5 min, `wgsync` shape).
  For each host whose descriptor is **enabled + provisioned** and whose **latest report**
  `pbs_dr.state` is a stuck state held across a **debounce** (≥2 distinct reports — so a box briefly
  `waiting_secret` between provision and its first consume is not touched): `waiting_secret` →
  **re-stage the stored secret** (no ep0 call, **no generation bump**); no stored secret →
  **escalate to Re-issue**; `consumed_failed` → **escalate to Re-issue only** (a re-stage would
  re-feed a burned secret). A converged/`disabled`/`verify_failed`/unprovisioned/DR-OFF host is a
  **pure no-op**. Each heal emits a distinct audit event (`pbsdr_selfheal_restaged` /
  `_reissued` / `_consumed_failed`). Reads the hub DB only — never the box.
- **`internal/store/pbsdr.go`:** `RestageHostPBSSecret(hostID) (bool, error)` — clears `consumed_at`
  IFF a row exists (no INSERT, no value change, **no generation bump**); `restaged=false` → the
  caller escalates. `PBSDRHealStates()` — one query joining each host's descriptor enable/provision
  flags to its latest report's `pbs_dr.state` + id (mirrors `GetHostOOBStates`).
- **`internal/web/pbsdr.go`:** `ReissuePBSDR(ctx, customerID)` — the non-HTTP core of the operator
  Re-issue button (tenantsync reissue → fresh consume-once secret → descriptor refresh + gen bump),
  now the reconciler's escalation seam. The operator handler is unchanged.
- **`cmd/hub/main.go`:** the reconciler is started unconditionally (the primary re-stage heal needs
  no endpoint). **`PBSDRHEAL_ONLY_HOST`** env scopes a supervised first rollout to one host
  (empty = whole fleet — the steady state).
- **Tests:** `internal/pbsdrheal/reconciler_test.go` (Scenarios A–F + scope + no-re-heal, real store
  + fake action seam) and `internal/store/pbsdr_test.go` (re-stage semantics + the **no-generation-
  bump** guard + `PBSDRHealStates` parsing). All six §10 red-proofs verified (mutation → FAIL →
  revert). No agent changes (the agent already self-heals once a secret is consumable).
- **Live-leg (drill guest qm300):** rolled back to `post_day0_golden136` (the re-install
  reproduction) → stuck `waiting_secret` (marker gone, `felhom-pbs` absent, hub secret consumed).
  Deployed scoped via `PBSDRHEAL_ONLY_HOST=demo-vm-felhom-2f4b00`. The reconciler observed
  `waiting_secret` across two reports (16:52 + 17:07 UTC) and at **17:10:00 re-staged** the stored
  secret (event `pbsdr_selfheal_restaged`; "no ep0 token minted, no generation bump"); the agent
  **re-consumed at 17:10:26** and **converged (`state=applied`) at 17:10:28** — hands-free, no
  operator click. The demo host `demo-felhom-01` was never touched (out of the scoped work set).
  **Fleet-wide widening (remove `PBSDRHEAL_ONLY_HOST`) is a deliberate operator follow-up** — not
  done in this supervised session.

## v0.55.0 — accept the offbox_enlarge_blocked event (Task 3a-fix delivery chain) (2026-07-15)

The controller (v0.134.1) sends an `offbox_enlarge_blocked` warning when an app's enlarged offsite
push is refused by the quota gate (config+DB still saved). This closes the **ingestion** link of its
delivery chain.

- **`internal/api/handler.go`:** `offbox_enlarge_blocked` added to `allowedEventTypes` — the event was
  400-rejected before (the customer email was dropped at ingestion). Acceptance case +
  400-red-proof added to `internal/api/event_test.go`.
- **Deliberate NON-change:** NO `customerMessages` entry. `FormatCustomerEmail` (`templates.go:129`)
  gives the static per-type message PRIORITY over the raw message, so a static entry would DISCARD
  the controller's dynamic two-number Hungarian text (estimate + quota). The documented fallback (raw
  message survives) is the correct path; locked by a new `internal/notify/templates_offbox_test.go`
  assertion. `internal/notify/dispatcher.go` and the customer `EnabledEvents` whitelist are unchanged
  here — the controller side (v0.134.1) owns `DefaultEnabledEvents` + the prefs migration + the
  settings checkbox.

## v0.54.0 — operator login password changeable from the UI (2026-07-13)

The hub login password was previously settable ONLY by editing the `auth.password_hash` field in
the `hub-config` ConfigMap and redeploying — there was no in-app way to change it. Added a
**"Login password"** card on the Configuration page.

- **DB-override precedence** (same pattern as the controller-version floor). New `hub_settings` key
  `operator_password_hash` (store: `Get/SetOperatorPasswordHash`). The web server no longer reads a
  static field for auth: `Server.passwordHash` is renamed `configPasswordHash` (the hub.yaml SEED)
  and every auth check — the CSRF gate, `RequireAuth` session/basic-auth paths, and `handleLogin` —
  now goes through `effectivePasswordHash()` = **DB override wins, else config seed**. The ConfigMap
  value stays the **break-glass fallback**: blank the DB row (or edit the manifest + redeploy) to
  reset a lost password.
- **`POST /configuration/password`** (`handleChangePassword`): requires the **current** password
  (verified against the effective hash), a new password of 8–72 bytes, and a matching confirmation;
  rejects a no-op change. On success it bcrypts the new password (cost 10, matching the seed) and
  persists the override. Existing sessions are intentionally kept valid — only the next sign-in and
  Basic-Auth use the new hash. CSRF-enforced (existing `ServeHTTP` gate); no secret is ever logged.
- **UI**: change-password card on `configuration.html` with current/new/confirm fields, inline
  client-side mismatch pre-check, and six flash outcomes (`pw_changed`, `pw_current_wrong`,
  `pw_too_short`, `pw_too_long`, `pw_mismatch`, `pw_unchanged`).
- **Tests + red-proofs** (`change_password_test.go`): override-wins precedence, happy-path
  end-to-end through `handleLogin` (new works, old dead), wrong-current rejection (security anchor),
  mismatch/too-short/no-op rejections, and template render. Red-proofs verified — dropping the
  current-password check writes the override anyway (WrongCurrentRejected fails); breaking the
  override precedence kills both the precedence and happy-path login assertions.

## (unreleased) hostInstallVersion 1.16.0 (2026-07-13)

Display-const bump only, keeping `scripts/hostinstall_gates.py` green with the installer's
v1.16.0 (FELHOM_ESCROW via the canonical sudoers fetch — see scripts/CHANGELOG.md). No behavior
change; rides the next hub image train (no deploy for this).

## v0.53.0 — closing bundle: F-14 gated auto-Reissue + dead-host roll-up honesty + bearer out of git (2026-07-13)

The last engineering items on the pre-tester board. Two operator rulings in force (CONTEXT.md):
F-14 auto-re-issue only on a recorded escrow-acked deletion; customer status never better than
its worst expected host.

- **F-14 deletion provenance + gated auto-Reissue** (take-two MEDIUM: host delete + re-enroll
  with a surviving ep0 tenancy = DR re-attach dead-end, `token_exists` on both auto-provision
  and config save). New `host_deletions` table — host_id, customer_id, deleted_at,
  `escrow_acked` (= ack given over a PRESENT escrow row) — written INSIDE the DeleteHost
  transaction; NO backfill (pre-record deletions keep the manual path by design). The provision
  atom, on `token_exists`, reads the customer's MOST RECENT deletion record: escrow_acked →
  invoke the EXISTING tenantsync Reissue op, store the `pbsdr_auto_reissue` audit event
  ("Previous key destroyed (acknowledged deletion) — credentials re-issued automatically."),
  proceed; no record / un-acked → the pre-existing refusal byte-unchanged (never-silently-re-key
  law). Red-proofs: provenance-write drop → scenario-A fails; gate bypass → scenario-B's
  zero-reissue assertions fail (silent re-key visible as a 303).
- **Dead-host roll-up honesty** (drill-1 observation, live on the Peti cluster: proxmox1 down
  23h behind a GREEN customer row — controller reports ride the internet, independent of the
  agent). Customer status on the dashboard, /configs list and customer detail (header + strip)
  is now `worst(controllerDerived, hostStatusOf(each expected host))` via THE single staleness
  definition (`Server.hostStatus`; no second threshold anywhere): any host down/stale caps the
  customer at WARN with a cause chip naming the host ("host down: <id>"); pending hosts worsen
  only after the customer has ever reported (onboarding exclusion). The three inlined
  controller-status chains collapsed into `controllerStatus()` (rollup.go). Display +
  derivation only — HostStalenessChecker alerting untouched. Red-proof: fold removal → the
  exact Peti fixture renders green → TestRollup_DeadHostMasking fails.
- **Operator bearer out of git** (the two publish runbooks' ROTATION item): `manifests/hub.yaml`
  no longer commits `report_api_key` — the Deployment injects `REPORT_API_KEY` from out-of-band
  `Secret/report-api` (deliberately NOT `optional:` — a missing Secret fails Ready instead of
  booting an unauthenticatable hub); main.go gains the env override (RESEND_API_KEY twin). New
  gate `scripts/manifest_bearer_gate.py` blocks bearer-shaped (64-hex) literals in manifests/
  (red-proven: reintroduction → exit 1). The controller repo's example-config copy of the
  literal is scrubbed. The exposed git-history value dies with the SUPERVISED rotation —
  procedure + full consumer list in documentation/runbooks/secrets.md §"Operator/global bearer
  key" (per-customer/per-host keys unaffected).

Hub half of the polish batch (take-two findings F-15/F-16). Companion: controller v0.123.0.

- **F-15 instant reset codes**: `POST /api/v1/claim/reset-request` now returns the ACTIVE code
  state in the response — `{claim: {code_hash, generation, issued_at}}`, the exact shape and
  bcrypt-only guarantee of the report ACK — so the box applies the rotated hash in the same
  request cycle and the emailed code works immediately (previously the box learned it only on
  its next report ACK, ~15 min — Viktor's take-two live failure). Served on every authorized
  outcome (a cap-reached refusal returns the unrotated row = controller-side no-op by
  generation). Never a plaintext code. Live-proven: apply 1 s after the request; code accepted
  on first try. The operator "Kód újraküldése" (no box round-trip) still has the ACK lag — its
  flash + data-confirm copy now say so ("A kód a doboz következő jelentésekor (~15 percen belül)
  aktiválódik."); the previously unmapped `claim-resent`/`claim-resend-failed` flashes render.
- **F-16 inline confirms**: every native `confirm()` in the hub UI (offsite re-issue, freeze/
  unfreeze, PBS re-issue, telemetry reset, dismiss-all-issues, regen-password, claim-resend,
  block/delete, geo-disable) replaced by the LIGHT inline two-step — the shared
  `inline_confirm.html` partial (`felhomConfirm` + `data-confirm` delegation, `requestSubmit` so
  formaction survives). Native confirms are OS-modals that froze CDP browser automation (F-16;
  the drill F-11 siblings). NOT the danger-zone typed-confirm — that heavyweight cascade flow is
  untouched. New gate `scripts/hub_confirm_gate.py` (zero native confirm/prompt; red-proven).
  Live-proven: the offsite re-issue completed under automation without freezing.

## v0.51.0 — DR-tier-by-default: per-customer flag + hands-free cascade + offsite coupling + capability chips (2026-07-12)

Hub half of the DR-tier-by-default batch (DRILL-day0-vm-2026-07-12; operator decisions 1–5:
capability BAKED on every install, activation is THIS flag, DR defaults ON for new customers,
identity-only escrow PARKED by policy, WG is base infrastructure). Companion: installer v1.15.0
+ agent v0.86.0 (capability `inactive` state).

- **Per-customer `dr_tier` flag** (customer_configs column + form checkbox in the renamed
  "DR tier (PBS, ep0)" section, replacing the old `pbsdr_enabled` form field). NEW customers
  default ON; legacy rows were initialized FROM REALITY by a one-time migration backfill
  (host carries an enabled pbs_dr descriptor → ON, else OFF — never auto-cascade a legacy box;
  backfill runs only on the ALTER that adds the column, so later operator opt-outs survive).
- **Cascade semantics** (scenario D): an UNMET precondition (no host / no WG peer / no
  tenantsync) is no longer a save-blocking error — the flag stores the intent and the edit form
  shows per-stage status (host enrolled → WG peer → descriptor provisioned → escrow present),
  reusing the fail-closed guard wording. REAL provisioning failures stay fail-closed (tenantsync
  error, token-exists → Re-issue).
- **Hands-free auto-provision** (scenario A): a host's FIRST WG peer registration fires
  `PBSDRAutoProvision` (api `SetWGRegisteredHook`, wired when tenantsync is enabled) — a DR-ON
  customer's descriptor provisions with ZERO operator steps and applies on the agent's next
  desired-state tick. Detached goroutine; never delays/fails the registration response.
- **Offsite requires the DR tier** (scenario C, drill F-6 CLOSED BY POLICY): `applyOffsite`
  refuses without the flag — exact message "Offsite backup requires the DR tier — enable it
  first (the escrow ceremony depends on the PBS key)". No more provisioning into the
  EscrowState-pending-forever dead end.
- **Capability chips on the host page** (NEW render surface): the agent's privileged-capability
  self-check is now visible — ok (blue), degraded (warn / error when critical), and the agent
  v0.86.0 `inactive` state as a NEUTRAL chip (disabled ≠ degraded). The pre-v1.15.0 pbsdr
  "binary not found" signature surfaces the migration one-liner (never silently pretend).
  `.badge-ok` finally defined in style.css (was referenced, fell back to bare `.badge`).
- Setup-tab installer copy: `hostInstallVersion` 1.12.0 → **1.15.0** (drill F-1), now gated
  against the installer's SCRIPT_VERSION by `scripts/hostinstall_gates.py`.
- Tests + red-proofs (all four mutations proven red): coupling gate (guard removed → refused
  case fails), flag default (default flipped → form test fails), backfill
  (enabled:false ignored → disabled case fails), auto-provision (hook unhooked → scenario A
  test fails); plus cascade-wait, chips render (inactive-neutral / degraded-stays / migration
  hint), and the one-time-backfill-survives-reopen case.

## v0.50.0 — customer-claim password arc: code engine + email + ACK/config delivery (2026-07-12)

Hub half of the customer-claim password gate (closes DRILL-day0-vm F-4/F-5; needs controller
v0.122.0). The customer OWNS the dashboard password — the hub generates a one-time claim code,
emails it (Hungarian) to the REGISTERED address, and stores only `bcrypt(code)`. No operator-set
path; the plaintext code exists solely inside the email send (the retrieval-passphrase custody
rule).

- **`internal/claim`** — the code engine. `EnsureIssued` (idempotent — issue+email at the FIRST
  real config retrieve = Day-0, and at a live box's first report; repeated pulls/reports never
  rotate or re-send), `Resend` (operator button; rotates generation — unclaimed gets the claim
  template, claimed gets the reset template), `RequestReset` (controller-forwarded "Elfelejtett
  jelszó", rate-limited 3/day/customer), `MarkClaimed` (set-only; one confirmation email on the
  unclaimed→claimed transition).
- **`store.customer_claims`** — per-customer `{code_hash, generation, issued_at, emailed_at,
  claimed_at, reset_day, reset_count}`. `RotateClaimCode` bumps the generation (single active
  code) and PRESERVES `claimed_at` (a reset never un-claims); `MarkClaimed` is set-only.
- **Delivery**: `GET /api/v1/config/{id}` bakes `web.claim_code_{hash,generation,issued_at}` into
  the generated controller.yaml (gate-from-first-boot) and issues the first code; the report ACK
  serves the active `claim {code_hash, generation, issued_at}` (allowlisted) and ingests the
  controller's `claimed` flag (set-only). `POST /api/v1/claim/reset-request` (self-scoped by the
  box's report key). New emails via the notify dispatcher; `claim_lockout` event allowlisted.
- **UI**: the customer page Setup tab shows a claim status chip (Nyitott — kód kiküldve / Claimed)
  + a "Kód újraküldése" button (`POST /configs/{id}/claim-resend`) — no plaintext code ever
  rendered (there is none to render).
- 15 tests (engine, ACK/config, UI); the arc's red-proofs live in the controller repo (gate) +
  here (generation bump, reset non-DoS).

## v0.49.0 — Edit tab merge (edit-a), scoped auto-refresh, style.css cache-bust (2026-07-12)

> The task spec targeted "v0.48.0", but v0.48.0 (app_start_failed, below) had already shipped +
> deployed by the time this train ran — a published tag is never re-pointed, so this is v0.49.0.
> Baseline `3e949bc`; commits `e740147` → `2e03de1` → `1d94b1a` → docs/manifest.

- **Edit tab merge (edit-a)** (`templates/customer_unified.html`, `templates/config_form.html`,
  new `templates/config_form_body.html`, `web/configs.go`, `web/pbsdr.go`): the standalone
  customer edit page merged into the customer page's Settings tab, renamed **Edit**. The form
  body is a shared `{{define "config_form_body"}}` sub-template (the `host_detail_body`
  pattern) built by the one `configFormData` view-model builder; the standalone chrome keeps
  rendering it for the create flow (`/configs/new`) and the validation-error re-render. The
  Edit tab renders: config form, Controller Update card, Geo card, and a **Danger zone** card
  holding the Block/Unblock/Delete forms relocated verbatim from the Customer Info header
  (endpoints + `confirm()` unchanged) — all SIBLINGS after `</form>` (nested forms are invalid
  HTML and would break the offsite/PBS `formaction` sub-buttons). The header keeps only the
  config-less Create Config action. `GET /configs/{id}/edit` → 302 `/customers/{id}#tab=edit`;
  tabs JS gains the `settings`→`edit` legacy-hash alias.
- **Server-side required fields on update** (`web/configs.go` `handleConfigUpdate`): the twin
  of the form's `required` attributes (Display Name + Domain), checked BEFORE provisioning; the
  error path re-renders the standalone page with the SUBMITTED overrides so typed values are
  never lost (red-proofed: nil overrides → values reset → test fails).
- **Redirect anchors**: update/block/unblock/offsite-reissue/offsite-freeze/pbsdr-reissue →
  `?flash=…#tab=edit`; regen-password → `#tab=setup` (its card lives there); delete unchanged
  (`/configs?flash=deleted`).
- **Scoped auto-refresh** (`templates/customer_unified.html`): the 60s reload fires only while
  a live tab (`data-live-tabs="overview,applications,events,host"` on the nav) is active AND no
  form is dirty (delegated document-level input/change listener, never reset — a reload clears
  it). Skipped ticks reschedule; a muted `(paused)` hint shows next to the toggle on non-live
  tabs / dirty forms. Toggle, `hub_auto_refresh` localStorage key, cadence, default-on: unchanged.
- **style.css cache-bust** (all `templates/*.html`): every stylesheet link is now
  `/style.css?v={{hubVersion}}` — closes the v0.47.0 gotcha (`max-age=3600` served stale
  styling for up to an hour after each deploy). Red-proofed (bare link fails the render test).
- **Repo staging rule** (`CLAUDE.md`): never `git add -A` in this repo (the v0.47.0 `146d165`
  sweep incident) — explicit paths, pull-rebase, one writing session per clone.
- Tests: +11 (Group A panel surface / sibling-form / header-count, Group B redirect + create +
  typed-values-preservation table, Group C refresh structural pins, Group D cache-bust sweep).
  Amended pins: `customer_tabs_test.go` (settings→edit), `pbsdr_test.go` (postUpdate supplies
  the now-required fields; FormRendersState asserts the embedded Edit-tab render).

## v0.48.0 — accept the app_start_failed event (controller fix-3, CAMPAIGN-3) (2026-07-12)

- `app_start_failed` added to `allowedEventTypes` (`internal/api/handler.go`) + `customerMessages`
  (`internal/notify/templates.go`). Without the allowlist entry the controller's fix-3 event (a
  DEPLOYED app found not running — controller v0.120.0) would 400 at ingest and never reach the
  operator. No other hub change; pairs with controller v0.120.0 which closes the CAMPAIGN-3 finding set.

## v0.47.0 — UI reorganization: customer tabs, Host tab, stale-host removal, offsite multi-endpoint UI, button contrast (2026-07-11)

Five hub-side deliverables; no agent/controller/protocol changes. Baseline `8e1a3f0`
(v0.46.0); commits `9f29bf3` → `ae950e5` → `146d165`(swept WIP) → `068427a` → `0daddcd`.

- **CSS button contrast** (`templates/style.css`): `.data-table td a` → `:not(.btn)` (base +
  hover) — `<a class="btn">` inside data-table cells (host-detail Diagnostics View/Download,
  customer log-tail buttons) rendered blue-bright on blue-bright, i.e. invisible. Plain table
  links keep the bright-link style; `.btn` itself untouched, no `!important`.
- **Customer page tabs** (`templates/customer_unified.html`, `style.css`): the ~18 stacked
  sections split into 8 client-side hash tabs (`#tab=` overview / applications / setup /
  settings / backup / events / notifications / host) + a sticky summary strip (name, status,
  controller version, last report, containers chip). Graceful degradation is load-bearing:
  panels hide only under a JS-added `body.js-tabs` class — no JS = every section visible, all
  existing render tests pass unmodified. Events tab carries a red error-count badge (reuses
  the already-fetched `CountEventsBySeverity` data — no new query). The auto-refresh reload
  preserves the hash → the active tab survives. No handler/data-model change for the tabs.
- **Host tab + shared sub-template** (`templates/host_detail_body.html`, `web/hosts.go`,
  `web/configs.go`, `store.ListHostsByCustomer`): the host-detail body extracted into a
  `{{define "host_detail_body"}}` rendered by BOTH `/hosts/{id}` (chrome + call) and the new
  per-customer Host tab (a LIST by design — 1 host today, N for a later HA cluster; empty
  state otherwise). `handleHostDetail`'s data assembly extracted into `hostDetailData`.
- **Stale host removal** (`store.CountHostArtifacts`/`DeleteHost`, `web/hosts.go` handlers,
  routes above the `/hosts/` catch-all): `GET /hosts/{id}/delete-impact` (counts/booleans
  ONLY) + `POST /hosts/{id}/delete` behind a type-to-confirm dialog (global-floor pattern).
  Gates: ONLINE host → 409 always (no override — a live agent would 401 forever; enroll is
  passphrase-gated mint-once); confirm mismatch → 400; escrow present without the explicit
  checkbox → 409 with the tx never started (`ErrHostEscrowPresent`, fail-safe-to-refuse).
  One transaction cascades guests, host_reports, signed_jobs, host_recovery,
  host_pbs_secrets, host-scoped log bundles (`scope_id == host_id` ONLY — customer-scoped
  bundles survive), the bound wg peer (inside the tx — no stranded peer on crash), escrow
  (only when acked), then the host row. The wgsync 5-min declarative push converges the
  endpoint afterwards — no reconciler change. Danger-zone card renders only when deletable,
  so the hosts-list zero-`<button>` pin and the detail-page 2-button pin stay green unmodified.
- **Offsite multi-endpoint UI** (`store/wg.go` `ListWGEndpoints`/`DeleteWGEndpoint`,
  `web/offsite.go`, `templates/offsite.html`): `/offsite` lists ALL `wg_endpoints` rows as
  cards + add/edit/delete forms (posture change from S2 read-only — operator decision).
  Validation → 400 stores nothing; subnet edit / endpoint delete refused 409 while peers sit
  in the (current) subnet; pubkey change gets a type-to-confirm noting pull-based
  convergence. Peer table gains an Endpoint column (first id-ordered subnet match, em dash
  when none). **Allocation, reconciler push and desired-state merge stay lowest-endpoint-id**
  (`GetWGEndpoint` untouched; the page states the deferral) — per-endpoint allocation
  (`wg_peers.endpoint_id` migration) is a future arc.
- **Tests**: +21 new/amended across web+store: tab render (no-JS completeness, badge, banner-
  above-tabs), Host tab shared-body/empty/isolation, delete cascade + refusal non-effects +
  impact shape + danger-card gating, offsite cards/column/validation/guards. Five companion
  red-proofs ran and FAILED as required (online gate, escrow ack, bundle scope, endpoint
  delete guard, subnet-change guard). Note: commit `146d165` (parallel session) swept the
  Part-4 WIP mid-red-proof — `068427a` restored the escrow-ack line.

## v0.46.0 — observability pass: per-box log pulls, bundle custody, TTL + secret gate (2026-07-11)

Hub third of the cross-repo observability task (agent v0.83.0 + controller v0.116.0): remote,
pull-only access to both box components' always-DEBUG capture rings — honest to the sovereignty
posture (the hub never connects in; the box pushes on its own cycle and its own log records the
pull, customer-visible).

- **Store** (`internal/store/logbundle.go` + schema): `log_bundle_requests` (one pending intent per
  scope+component; scope = customer_id for the controller/report channel, host_id for the agent/
  heartbeat channel) + `log_bundles` (gzip payload, newest-3 retention, **72 h TTL** purged on the
  existing 60 s sweep). `SaveLogBundle` runs the **token-pattern secret gate BEFORE storing** —
  a hit stores a `blocked: possible secret` flag row with NO payload (fail-closed; WARN logged
  hub-side); `[REDACTED]` shapes and public checksums deliberately pass (red-proof: gate disabled →
  the planted `re_…` token stores → FAIL).
- **Channels** (additive, both directions): the report ACK gains `controller_log_requested` and
  ingests `controller_log_tail`; the heartbeat envelope gains `log_tail_requested` and ingests
  `log_tail`. Consume-once on arrival (red-proof: clear-on-arrival removed → the ACK re-advertises
  forever → round-trip tests FAIL). A pre-0.83 agent simply never fulfills — the request stays
  visibly `pending` (S6 tested), harmless.
- **UI** (host detail, English like the rest of the hub operator surface): a Diagnostics section
  with **Request controller logs / Request agent logs** buttons (CSRF form posts), state rows
  (`pending` with the honest per-channel latency hint — controller ≤ ~15 min report interval,
  agent ≈ heartbeat cadence — / `available` with View+Download / `blocked`), and the 72 h custody
  note. The hosts read-only invariant is amended: these two request forms are the ONLY actions
  (pinned by test).
- Download endpoint `/hosts/{id}/log-bundles/{bid}[?download=1]` — session-authed like the rest of
  the operator UI, scoped to the host's own channel scopes.

## v0.45.0 — floor-UI separation + effective-floor source + per-box MinAgent conditional floor (2026-07-11)

Two parts of the NAS/coupling backlog, both addressing the publish-train 0.81/0.113 floor footguns.

- **Floor-UI separation + confirm (Part C):** the global controller-version floor is its own card with
  a **type-to-confirm** dialog that first states the live blast radius —
  `GET /configuration/global-floor/impact?v=X.Y.Z` (`countBoxesBelowFloor`, honoring per-customer
  overrides) — so "the floor acts immediately" is impossible to miss. An **effective-floor + source
  line** (`store.ResolveGlobalFloor` → `GlobalFloorResolution`) shows the resolved value and WHICH
  source won (**DB `hub_settings`** vs **env `DEFAULT_MIN_CONTROLLER_VERSION`**, both raw values when
  they differ) — the 9-minute-skew incident's root cause, now permanently visible. The Day-0 artifact
  manifest save is unchanged and provably does NOT touch the floor (regression-tested).
- **Per-box MinAgent conditional floor (Part D):** the artifact manifest gains **`MinAgent`** (the
  golden's controller `MinAgent:` header; blank = uncoupled, no gating). At report-ACK time
  `store.ResolveManagedFloor(customerID)` compares the box's `hosts.agent_version` against it: agent ≥
  MinAgent → the controller floor is served; below **or unknown** → the floor is **HELD** (ACK omits
  the directive) and the box is flagged on the Hosts dashboard (`floor held: agent <v> < MinAgent
  <w>`). Mechanises the "agent BEFORE controller floor" rule per box — the manual fleet check is
  retired (publish-train-rules.md rule 3 updated).
- **THE one comparator:** `web.compareVersions`'s body moves to a leaf `internal/semver` package
  (`Compare`/`Valid`); web delegates, store's MinAgent gate reuses it (no import cycle, no second
  comparator; gitea's documented local copy is out of scope).
- Tests + red-proofs: floor source precedence (DB-wins), manifest-save-doesn't-touch-floor,
  impact-count with override exclusion, source-line render; managed-floor hold/serve/uncoupled/
  unknown-agent + a fleet discriminator + the report-ACK wire test (held box omits the floor, served
  once the agent qualifies). Every red-proof run → predicted failure → reverted.

## v0.44.0 — PBS DR tier SLICE 1: ep0 tenantsync surface + hub provisioning (2026-07-10)

Builds on SPIKE-pbs-tier-provisioning (00afadc). The operator ticks "PBS DR tier (ep0)" on a
customer config → the hub verifies the host's WG peer (the agent self-registers it; absence is
fail-closed) → provisions the per-customer ep0 PBS namespace + privilege-separated token over the
NEW `felhom-tenantsync` forced-command surface (peersync untouched) → stores the token secret
CONSUME-ONCE, host-scoped → serves the non-secret descriptor via the host desired-state (generation
bump). The agent apply-bridge is SLICE 2 — nothing is live-provisioned yet.

- **`scripts/felhom-tenantsync.sh` v1.0.0** (installed on ep0 per runbook §10): JSON-on-stdin/stdout;
  ops `provision` (existing token = hard error `token_exists` — re-issue is explicit), `reissue`
  (delete-token purges ACLs → recreate → re-grant), `fingerprint`. Dual-grant per spike §3;
  own-namespace self-check with one regen retry (the spike's transient-403 note) then rollback.
  Secret hygiene: the token secret rides stdout ONLY (all tool stdout → stderr; never a file/argv).
  NO deprovision op — namespace/data deletion stays a deliberate, separate decision.
- **`internal/tenantsync`**: the wgsync twin — pinned host key (exact-match, constrained
  HostKeyAlgorithms), per-op JSON exec, typed `ErrTokenExists`. Divergence from wgsync: error
  messages NEVER embed stdout (the secret channel) — red-proof-style contract test
  (`TestErrors_NeverEmbedStdout`).
- **Store**: `host_pbs_secrets` (host-scoped consume-once, the `one_time_secrets` twin) +
  `SaveHostPBSSecret`/`ConsumeHostPBSSecret` (same-tx mark; re-save resets).
- **API**: `POST /api/v1/hosts/{id}/pbs/consume-token` — per-host key, self-scoped (global key =
  operator recovery); 200 exactly once → 404; a foreign key's 403 does NOT burn the secret.
  (Task spec wrote `/host/{id}/…`; implemented under `/hosts/` for namespace consistency with
  every other agent-facing route.)
- **Web**: config-form section "PBS DR tier (ep0)" (enable + storage-id, default `felhom-pbs` —
  the descriptor carries the id so the slice-2 bridge is name-agnostic and the demo's
  `felhom-offsite` adoption dissolves the naming collision) + provisioned line + **Re-issue PBS
  credentials** (the offsite F4 precedent). `applyPBSDR` merges the non-secret `pbs_dr` descriptor
  into the HOST `desired_json` (admin-set path, `SetHostDesired` bump) — ConfigJSON never carries
  it. Fail-closed on: no tenantsync key, no enrolled host, no WG peer, no endpoint record,
  tenantsync error, `token_exists` (message points at Re-issue). Already-provisioned re-save =
  success-no-op (no re-key, no second secret, no spurious bump). Disable = descriptor
  `enabled:false`, tenancy kept.
- **Deploy**: `manifests/hub.yaml` gains `TENANTSYNC_SSH_KEY_FILE` + optional `Secret/tenantsync`
  mount (same endpoint addr + pinned host key as peersync, its own key).
- Red-proofs run and recorded (REPORT.md): consume-once mark drop → the secret re-serves (store +
  API layers); fail-closed guard swallow → 303 half-save; idempotency short-circuit drop →
  token rotation + fresh secret + spurious bump.

## v0.43.1 — Git Sync form hint: credentials are optional (2026-07-10)

Pairs with controller v0.112.0 (anonymous registry self-update). The config editor's Git Sync section
looked load-bearing; in truth the credentials matter only for a private app catalog — version discovery
and self-update work without them since controller v0.112.0. One template hint added
(`config_form.html`): "Opcionális — csak privát alkalmazás-katalógushoz. A verziófrissítés enélkül is
működik." No behavior change.

## v0.43.0 — Remote app-log diagnostics: copyable issues + error context + on-demand log tails (2026-07-10)

Pairs with controller v0.111.0. Motivated live: Peti's CWA NFS issue was visible in Known Issues but
tooltip-only unreadable and context-free, and there was no way to see the app's actual logs without box
access. The hub still never connects into a guest — everything rides the existing report + ACK.

- **Part A — readable, copyable issues** (`templates/app_detail.html`): Known Issues rows are
  click-to-expand — full message in a wrapping monospace `<pre>` + **Copy** button (clipboard API with
  execCommand fallback), fingerprint/severity/first-last-seen in the body. Tooltip-only truncation killed.
- **Part C — context stored + rendered** (`store/telemetry.go`): `app_log_issues` gains `context`
  (JSON array) + `context_customer` (provenance); `upsertAppIssue` stores context on INSERT and adopts a
  later one ONLY while the stored context is empty (**first capture wins** — stable repro, no churn).
  Rendered in the expanded row as "Context around first occurrence — from <customer>", copyable.
  Nil-safe with pre-v0.111 reports.
- **Part D — on-demand ordered log tail (pull-based)**: per-app **"Request log tail"** button on the
  customer page → `log_tail_requests` row (one active per app; re-click refreshes) + a customer-visible
  `log_tail_requested` event (transparency by default). The report ACK advertises
  `log_tail_requests: [app…]` (same additive omit-when-empty pattern as escrow); the controller's next
  report ships `log_tails` → stored in `app_log_tails` (transient, **last 2 per app kept**) and the
  request is **cleared (consume-once)**. Ordered tail view with line numbers (`log_tail.html`) +
  **Download .log**; tail reads are customer-scoped.
- **Part F fix — the 24h/7d/30d selector now filters Known Issues**: `GetAppIssues` gained the same
  `since` cutoff the Memory Trend uses (it had NO time filter — a 24h view showed 25-day-old rows).
- **Part G — deletion → dismissal**: diagnosis = the delete handler was NOT broken; deletion is futile
  because the controller re-scans its rolling 15-minute window every report and re-upserts a still-
  occurring fingerprint with fresh `last_seen` minutes later. Replaced with `dismissed_at`:
  Dismiss Selected/All (buttons renamed), dismissed rows out of the default view ("Show dismissed"
  toggle), and `upsertAppIssue` un-dismisses ONLY on `excluded.last_seen > dismissed_at` — a re-sent
  old window stays hidden, a genuinely NEW occurrence resurfaces (recurrence never silently swallowed).
- **Part H — per-customer scoping**: `?customer=<id>` on the app detail page filters Known Issues to
  rows whose `affected_customers` contains the id (header shows "filtered: <customer>"); the customer
  page's App Telemetry rows link there (the drill-down). The fleet view stays; the expanded row lists
  the affected customers explicitly (linked) and the count column is labeled
  "Occurrences (all customers)".
- **Tests + red-proofs (all four failed exactly as designed, restored green):** dismissal guard dropped
  → old-window re-report resurrected the row → FAIL; range predicate neutered → 10d-old issue visible
  at 24h → FAIL; first-capture-wins dropped → empty-context upsert clobbered stored context → FAIL;
  consume-once DELETE removed → request survived fulfillment (store test + API ACK round-trip both) →
  FAIL. Plus: late-context adoption, warn-no-context, occurrence counting, tail request/fulfill/prune-
  to-2/cross-customer-404, ACK omit-when-empty baseline, render tests (expanded row content, customer
  page sections, ordered tail view + download headers).

## v0.42.0 — Remote "Debug mód" toggle on the customer config editor (2026-07-10)

Lets an operator flip the controller's debug mode (verbose log + the `/debug` menu, which the controller
gates on `Logging.Level=="debug"` / `isDebug()`) **remotely, without SSH** — the support workflow (today:
Peti's box). The config-version bump on save makes the controller re-pull + self-restart on its next
report ACK, so the switch takes effect within a cycle, hands-free.

- **Form field, not raw-JSON injection — on purpose.** `handleConfigUpdate` REBUILDS `ConfigJSON` from the
  form on every save (`buildConfigJSON`), so any foreign key injected straight into the stored JSON is
  dropped on the next save. The toggle is therefore a real form field, which by definition survives every
  save. (The offsite descriptor survives via its own separate provision-merge, untouched by this.)
- **`buildConfigJSON`** (`internal/web/configs.go`): `debug_mode` checked → emits `"logging":{"level":"debug"}`;
  unchecked → the `logging` key is OMITTED entirely (the generated `controller.yaml` default stands — no
  needless `"info"`).
- **Config form** (`templates/config_form.html`): new collapsible **"Hibakeresési mód (fejlesztői)"** section
  with the `debug_mode` checkbox; render state parsed back from `ConfigJSON` (`logging.level=="debug"` → checked).
  No change to the offsite/CF/git leg; no generic raw-JSON editor (deliberately — validated surfaces only).
- **Tests (non-hollow, `configs_debug_test.go`):** form→JSON both ways (checked emits / unchecked omits);
  full-path **survival test** through `handleConfigUpdate` proving the debug key lands, the offsite descriptor
  is byte-for-byte unchanged across save+re-provision, and the **red-proof** that a hand-injected foreign key
  is gone after one save (why the switch must be a form field); render state both ways. Red-proof exercised
  (feature disabled → survival + form tests fail).

## v0.41.0 — SLICE 4: OffsiteChecker (fill + staleness) + operator freeze lever (2026-07-09)

The last build item of the offsite arc (pairs with controller v0.109.0's soft-quota gate + report object).

- **`internal/monitor.OffsiteChecker`** — a SIBLING of StorageFillChecker (same born/persistent,
  escalation-only, recovery-re-arm shape; NOT bolted onto the disk checkers), reading the controller
  report's new `offsite` object. Two signals: **fill** (`repo_size_bytes` vs `quota_gb` at warn 90 / crit
  95 — quota 0 = dedicated, never alerts) and **staleness** (`offsite_stale`, warning): enabled+**escrowed**
  but no run in >48h (or never) — the silently-STUCK detector; a recently-FAILING offsite is not stale
  (`backup_failed` owns that), and pending/disabled targets never alert (normal onboarding — **companion
  red-proof:** dropped the escrowed-only filter → the pending customer alerted → test FAILED). Nil-safe on
  reports without the object (pre-v0.109 controllers). Tie-guard: duplicate same-second latest reports are
  processed once per sweep. Same 60s sweep as the other checkers.
- **Freeze lever (operator, MANUAL only):** `Provisioner.SetOffsiteFrozen` — flips ONLY `readonly` on the
  exactly-1 labelled sub-account via `UpdateSubaccountAccess` (SSH stays on; ambiguity refuses — tested),
  wired to confirm-gated **Freeze/Unfreeze offsite** buttons next to Re-issue (shared model only; dedicated
  is Hetzner-enforced). NEVER automatic — freezing also blocks prune, the customer's only way DOWN from
  over-quota. Route `POST /configs/{id}/offsite-freeze` (`unfreeze=1` reverses); action logged, value-free.

## v0.40.0 — SLICE 3: store the escrow password-hash + serve escrow status in the report ACK (2026-07-09)

The hub-verified escrow auto-confirm chain, hub third (pairs with agent v0.79.0 + controller v0.108.0).
The controller must verify the RIGHT fact — not "a blob exists" but "the blob covers the CURRENT repo
password" — so the hub records WHICH password each escrow covers, as a non-reversible sha256 (a 256-bit
random secret's hash is safe to store/serve; the password itself never reaches the hub).

- `internal/store`: additive migration `ALTER TABLE host_escrow ADD COLUMN restic_pw_sha256 TEXT`
  (NULL on legacy rows — e.g. the demo's — which therefore never auto-confirm; the deprecated manual
  confirm covers them). `HostEscrow.ResticPwSHA256` + `SaveHostEscrow` gains the param (last-write-wins);
  NULL-safe reads via COALESCE. New `GetEscrowStatusForCustomer` (hosts⋈host_escrow; latest-updated wins).
- `internal/api`: `escrowUploadRequest.restic_pw_sha256,omitempty` (the agent emit struct's mirror —
  `TestEscrowUploadContract` updated in lockstep with the agent's half); stored on upload. The **report ACK**
  gains `escrow: {identity_blob_present, restic_pw_sha256, created_at}` — omitted entirely when the customer
  has no escrow row (a fresh customer stays pending silently).
- Tests: hash stored + legacy-upload reads back NULL-safe as ""; ACK carries the object / omits it without a
  row; contract mirror.

## v0.39.0 — offsite hardening: F4 credential re-issue + F2 scan retry + F5 save UX (2026-07-09)

Part of the offsite-provisioning hardening bundle (pairs with controller v0.107.0 + agent v0.78.0); the
sharp edges from the live e2e (`documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`).

- **F4 (pilot-gating) — "Re-issue offsite credentials":** `Provisioner.ReissueCredentials` — the EXPLICIT
  operator recovery for a consumed-password dead-end (fresh-guest DR; consumed-but-failed install). Resets
  the customer's sub-account password (`ResetSubaccountPassword`) or dedicated-box password (new
  `ResetBoxPassword` in `hetznerapi`, client+interface+fake) → stores a FRESH one-time secret → the handler
  re-saves the config unchanged so `ConfigVersion` bumps and the stuck guest's next refresh re-runs the
  bridge. **Hard-scoped:** targets ONLY the resource labelled `felhom-customer=<id>`; refuses unless the
  label lookup finds exactly 1 (ambiguity = refuse, no reset, no secret) **+ companion red-proof** (dropped
  the exactly-1 guard → ambiguous lookup proceeded → test FAILED). NOT implicit rotation — `ProvisionOffsite`
  never calls it. UI: a confirm-gated button on the config form (shown only when provisioned), route
  `POST /configs/{id}/offsite-reissue` (CSRF rides the parent form). The password value is never logged.
- **F2 — host-key scan retry-with-backoff:** a fresh sub-account's DNS lags creation, so the FIRST save
  502'd (`no such host`, live). `scanWithRetry` retries on failure (default ladder 2/4/8/16/30s ≈ 60s total,
  inside applyOffsite's 3-min detached ctx; ctx-abortable; fail-closed past the budget) **+ companion
  red-proof** (disabled the retry loop → DNS-lag save failed → test FAILED). `Provisioner.ScanBackoff`
  injectable for tests.
- **F5 — save UX:** the config form disables its submit buttons and shows an in-flight notice on submit
  (the ~25–60s spinner-less save was the re-click bait that caused F1 live).

## v0.38.1 — offsite provisioning: detach from the client's request context (live finding F1) (2026-07-09)

Found in the first supervised live run: the offsite save takes ~25s (create + wait + host-key scan) with no
UI feedback, the operator re-clicked, the browser abandoned the first request, and `r.Context()` was canceled
**between `CreateSubaccount` and `SaveOneTimeSecret`** — the sub-account was created on Hetzner but its
one-time password was lost forever (the controller's consume 404s permanently; stranded resource).

- `internal/web.applyOffsite`: provisioning now runs on `context.WithoutCancel(r.Context())` with a 3-minute
  absolute timeout — once the create starts, the create→wait→store atom runs to completion even if the client
  disconnects. Fail-closed behavior unchanged (an actual provisioning error still 502s and saves nothing).
- Test `TestApplyOffsite_ClientDisconnectMidProvision` (a ctx-honoring fake cancels the request context
  mid-create): the one-time password must reach the store and the descriptor must merge despite the
  disconnect. **Companion red-proof:** reverted to the raw request ctx → the exact live error
  (`subaccount create action: context canceled`) → test FAILED. Restored.
- Known residuals (recorded, not fixed here): the form has no in-flight spinner/disable (the re-click bait),
  and a concurrent save can still hit Hetzner's box-level HTTP 423 action lock (surfaces as the fail-closed 502).

## v0.38.0 — offsite provisioning SLICE 2 (hub side): capture the box host-key fingerprint (2026-07-09)

Pairs with controller v0.106.0. So the controller can VERIFY the box identity instead of blind-TOFU, the hub
captures the box's SSH host-key fingerprint at provision and serves it in the descriptor.

- `internal/offsite`: `Descriptor.HostFingerprint` (SHA256:…, non-secret). `ProvisionOffsite` now captures it
  after the resource is ready via a `HostKeyScanner` seam (`SSHHostKeyScanner`, x/crypto/ssh — dials port 23
  and grabs the host key from the handshake, no ssh binary needed). **Fail-closed:** a nil scanner or a scan
  failure returns an error (don't serve a descriptor the controller can't verify). The controller re-scans and
  refuses on mismatch (v0.106.0).
- Tests: descriptor carries the fingerprint from a faked scanner; a scan failure fails-closed.

## v0.37.0 — offsite provisioning SLICE 1: Hetzner Cloud-API client + provisioning core (2026-07-09)

Slice 1 of the offsite-provisioning epic. On operator enable, the hub provisions a Hetzner storage-box
sub-account (shared) or dedicated box, generates the transient password, stores it one-time-consumable, and
serves the non-secret target descriptor to the controller via `ConfigJSON`. Coded against the API shapes
measured live in `documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (996d403). The
controller apply-bridge (SLICE 2), escrow auto-confirm (SLICE 3), and soft-quota enforcement (SLICE 4) are
separate slices.

- **`internal/hetznerapi`:** a typed client for the storage-box surface at **`https://api.hetzner.com/v1`**
  (NOT `api.hetzner.cloud` — the classic Cloud API 404s for storage boxes). Sub-account + box
  create/reset/access/change_type/delete/list-by-label + `WaitAction` (poll to `success`, bounded). A
  `CloudAPI` interface + an exported `Fake` so provisioning is unit-tested with **no live Hetzner calls**.
  Bearer token from an injected func (out-of-band secret; never logged).
- **`internal/offsite`:** `Provisioner.ProvisionOffsite` — **idempotent by `label_selector`
  (felhom-customer=<id>)** (names aren't unique); shared → sub-account on the pool box, dedicated → box;
  generates a 4-class transient password → `WaitAction` → `Store.SaveOneTimeSecret` → builds the NON-SECRET
  `Descriptor{enabled,type,host,user,port:23,repo_path:/home/felhom-repo, quota_gb|box_type}`. **Fail-closed:**
  any API/action error returns without a provisioned resource, one-time password, or descriptor.
  `MergeDescriptor` merges it under the `offsite` key of `ConfigJSON` (never a secret).
- **`internal/store`:** `one_time_secrets` table + `SaveOneTimeSecret` / `ConsumeOneTimeSecret` (single-use,
  return-and-mark in one tx). The transient password NEVER rides `ConfigJSON`.
- **`internal/api`:** `POST /offsite/consume-password/{id}` — serves the one-time password to the
  authenticated customer (same API-key auth as config-pull) EXACTLY once, then 404s. Never logged.
- **`internal/web`:** the config form gains an **Offsite backup** section (enable / type / soft-quota / box
  type); save → `applyOffsite` provisions (fail-closed: a provisioning error returns 502 and does NOT save)
  and merges the descriptor → `ConfigVersion` bump → controller re-pulls. Optional dep
  (`SetOffsiteProvisioner`), wired in `cmd/hub/main.go` from `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID`/`HETZNER_LOCATION`.
- Tests (faked Cloud API, no live calls): shared/dedicated provision + descriptor + one-time-password-stored
  + **password-absent-from-ConfigJSON**; idempotent re-save (no 2nd resource); fail-closed **+ companion
  red-proof** (swallow the create error → offsite marked enabled despite failure → test fails); one-time
  consume-once; `WaitAction` success/error/timeout; the consume endpoint (auth + single-use).
- **NOT yet live-provisioned** — awaiting the **dedicated-project scoped token** (the current token can delete
  ep0 — SPIKE §6); a live create is a supervised validation. Unit tests are this slice's proof.

## v0.36.0 — customer page: passphrase hardening + interactive install-command generator (TASK GL-7) (2026-07-09)

Two coupled, security-first changes to the operator-facing customer page (`customer_unified.html` +
`configs.go`). felhom.eu only; agent + host-install untouched.

- **Passphrase hardening (ships the security win).** The per-customer retrieval passphrase was
  rendered in cleartext twice — the visible `#retrieval-pw` node and baked into the Option-3 debug
  curl's `X-Retrieval-Password:` header. Now: `#retrieval-pw` renders a masked bullet run by default
  with **reveal** (`toggleSecret`) + **copy** (`copySecret`) controls, the value carried in
  `data-secret` (the existing reveal model). The Option-3 command carries a
  `<YOUR-RETRIEVAL-PASSWORD>` placeholder — the secret is NEVER in a copyable command block. (A
  zero-secret-in-DOM reveal-on-demand fetch is a deliberate future follow-up, not this task.)
- **Interactive install-command generator.** The three hard-coded install `<code>` blocks became a
  client-side builder (vanilla JS — no framework, CDN, or network) that assembles a live-updating
  command from form controls, emitting ONLY real host-install **v1.12.0** flags in a
  download-then-run shape (never `curl | bash`). CustomerID is prefilled from the server
  (`ScriptVersion`/`data-customer-id` via `pageData`); a byo selection requires `--cores`/`--memory`
  (enforced client-side with a `.gen-req`/`gen-msg` prompt); caps/mode are placeholders, never silent
  defaults. Graceful **JS-off static fallback**: the Option-1/2 code nodes keep a
  `--customer-id … --mode <appliance|byo>` command. The curated control surface excludes the seven
  dangerous/operator-only flags (`--force`, `--rotate-recovery`, `--enable-oob`, `--remove-golden`,
  `--uninstall`, `--adopt-pool`, `--rescope-acl`) — they are never offered as controls.
- **`configs.go`:** `const hostInstallVersion = "1.12.0"`; `pageData.ScriptVersion` added + populated.
- **Tests** (`render_test.go`): `TestTemplates_PassphraseHardened` (secret NOT in the Option-3
  command, placeholder present, masked-by-default bullet run, `data-secret` populated, reveal/copy
  controls present; red-proof = revert Option-3 to the raw secret → fails) and
  `TestTemplates_InstallGenerator` (all curated control ids present, script version + `data-customer-id`
  rendered, static-fallback command present, and none of the seven excluded flags appear page-wide).
- **Style** (`style.css`): `.gen-controls`/`.gen-radios`/`.gen-radio`/`.gen-check(s)`/number inputs/
  `.gen-msg` — dark palette, 2px radius.

## v0.35.0 — OOB operator access: operator peer + oob_peer_ip/oob_operator_ssh_key + OOB health alert (TASK H1) (2026-07-05)

The hub half of the merged E1+H1 operator-SSH-access feature (agent half = felhom-agent v0.72.0).

- **Operator OOB peer** (`store/wg_operator.go`): the fleet operator peer as an UNBOUND wg_peers row
  (host_id '', note operator-oob) at an EXPLICIT /32 (so the endpoint's static forward chain can
  hardcode it); validated in-subnet/not-reserved/not-taken; last-write-wins rotation. It rides
  ListWGPeers → peersync pushes it to the endpoint. `PUT/GET /admin/wg/operator-peer` (global key);
  the PUT also takes an optional `ssh_pubkey` (the operator authorized_keys line, hub_settings) and
  bumps EVERY host's generation.
- **Desired-state** (`api/wg.go` mergeWireguard): when an operator peer exists, the served wireguard
  block carries `oob_peer_ip` (rendered into the box's AllowedIPs — survives self-heal [OF-1]) and
  `oob_operator_ssh_key` (agent writes felhom-sshd's authorized_keys). Absent → byte-identical.
- **OOB health** (`monitor/host_oob.go`): ingests the agent's `oob` heartbeat stanza and raises a
  transition-based `oob_degraded`/`oob_recovered` warning (felhom-sshd down while the operator peer
  is configured, OR config invalid) — the proactive "can the operator get in right now" signal.

## v0.34.1 — mgmt_plane_healed alerts on the FIRST auto-heal (TASK G1 fix) (2026-07-05)

The mgmt-plane checker seeded a heal marker silently on first observation (copied from HostLeafChecker's
trust-on-first-report). A heal is an EVENT, not a baseline: construction still seeds pre-existing markers
(startup false-alarm guard), but a newly-observed marker now raises the warning — so the FIRST auto-heal
surfaces, matching the live drill. Added tests for both halves.

## v0.34.0 — break-glass recovery vault + mgmt_plane surfacing (TASK G1) (2026-07-05)

The hub half of the management-plane break-glass system (prerequisite for felhom-sshd / H1; agent half
= felhom-agent v0.71.0). Closes the recovery gap from
`documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8/#9.

- **Break-glass credential vault** (`store.host_recovery` + `internal/store/host_recovery.go`): a
  per-host root@pam console password, stored at rest, operator-retrievable — the human fallback for
  reaching the PVE web console (pveproxy, a failure domain distinct from sshd) when both the sshd path
  and the agent-independent auto-heal have failed. `PUT /hosts/{id}/recovery-credential` (SELF-scoped
  host key — day-0 vaults it) + `GET /admin/hosts/{id}/recovery-credential` (GLOBAL key only — a host
  key cannot read its own console password back). Secret discipline: never logged (username + length
  only); red-proofed that the password never reaches the hub log.
- **mgmt_plane surfacing** (`internal/monitor/host_mgmtplane.go`, on the 60s sweep): parses the agent's
  additive `mgmt_plane` heartbeat stanza and raises a `mgmt_plane_healed` WARNING when the watchdog
  auto-healed a missing `/run/sshd` (new `privsep_healed_at`) — a recurring clobber surfaces BEFORE it
  becomes a lockout, complementing host_staleness. Trust-on-first-report (seed, then alert on change),
  mirroring HostLeafChecker.

## v0.33.0 — S2 offsite connectivity: box-facing WG registration + wireguard desired-state block + /offsite UI (2026-07-04)

Doc 06 roadmap row S2 (commits `fcf84a0`/`ba52005`/`13203c2`); the S2 architectural decision:
the stored `desired_json` stays a pure OPERATOR blob — the WG assignment is HUB-owned state,
**merged into the served desired-state at READ time**, never written into the store.

- **Store** (`internal/store/wg.go` + `store.go`): `RegisterWGPeerForHost` (idempotent /
  **re-key-in-place keeps the /32** — stable tunnel addressing across rotation/DR / adopt-unbound
  S1 rows / typed `ErrWGPubkeyBoundElsewhere` — a key is never silently stolen); partial unique
  index `idx_wg_peers_host` = one bound peer per host; `BumpHostDesired` bumps ONLY the
  generation (the merge changes served state, not the blob); `allocateWGPeerTx` extracted from
  the S1 path behavior-neutrally (S1 tests unmodified). `WGPeer` gains `CreatedAt`.
- **API** (`internal/api/wg.go` + `handler.go`): `POST /hosts/{id}/wg` — per-host key
  SELF-SCOPED (global = operator/DR path); generation bump + endpoint push ONLY on real change
  (idempotent re-register moves nothing — asserted negatives). `mergeWireguard` injects
  `{endpoint{dns_name,wg_port,server_pubkey,pbs_tunnel_ip}, pubkey, assigned_ip}` into served
  desired-state; no peer → **byte-identical pass-through** (the cross-repo golden test passes
  UNMODIFIED); any merge failure → fail-safe unmerged serve (never 500 the control channel).
  `handleAdminSetDesiredState` REJECTS a top-level `wireguard` key (400 — an operator
  copy-paste-PUT can never clobber the hub-owned block). Admin DELETE of a BOUND peer bumps the
  owning host; unbound deletes move no generation. NEW golden
  `testdata/desired-state-wireguard.golden.json` = **the S3 cross-repo contract** (agent copy
  must stay byte-identical).
- **UI** (`internal/web/offsite.go` + `templates/offsite.html`): read-only `/offsite` page —
  endpoint card + peer table (truncated pubkeys, full value in title; bound peers link to
  `/hosts/<id>`); Offsite nav link in all 9 page templates. Mutations stay on the admin API
  (UI actions arrive with tunnel health, S3/S6).
- **Tests:** Groups A/B/C; five red-proofs run + reverted (self-scope drop, unconditional
  merge, rejection drop, bump-on-idempotent, script exit-swallow — see scripts/CHANGELOG
  v1.0.1). Old-agent (v0.63.0) tolerance proven live against the real felhom-pve record.

## v0.32.0 + v0.32.1 — S1 offsite connectivity: WG endpoint record + peer registry + pinned-SSH peer-sync (2026-07-04)

The hub side of doc 06's roadmap row S1 (`documentation/architecture/06-offsite-connectivity.md`),
resolving the slice-1 design point: **peer-sync = hub pushes over SSH** to a forced-command
reconcile script on the endpoint (pull/signed-manifest rejected — weakens immediate revocation;
HTTPS push API rejected — a new versioned binary + third public port for nothing).

- **Store** (`internal/store/wg.go` + migration in `store.go`, commit `b18f6ae`): `wg_endpoints`
  (single expected row "ep0") + `wg_peers` (presence = desired state; no status column — that's
  the S2 host-join). `AddWGPeer` = one tx, idempotent on pubkey, lowest-free-host `/32` allocation
  skipping network/`pbs_tunnel_ip`/broadcast, `UNIQUE(assigned_ip)` race backstop + one internal
  retry; typed `ErrWGEndpointUnset`/`ErrWGSubnetExhausted`.
- **wgsync** (`internal/wgsync/`, commits `fbeeacb` + `0fa7ea1`): `x/crypto/ssh` push client with
  `ssh.FixedHostKey` pin (no insecure fallback, ever) **+ `HostKeyAlgorithms` constrained to the
  pinned key's type** — the live validation caught a stock multi-hostkey sshd presenting ECDSA
  against the ed25519 pin (legitimate server refused); regression-tested with an in-process
  dual-hostkey SSH server. Reconciler pushes the FULL peer list (never deltas — drift repair by
  construction) on `Trigger()` or a 5-min tick; payload
  `{"version":1,"interface":"wg0","peers":[{pubkey, allowed_ip}]}`, deterministic order.
- **API** (`internal/api/wg.go`): `PUT/GET /admin/wg/endpoint`, `POST/DELETE/GET /admin/wg/peers`
  — GLOBAL key only (the `handleAdminSetDesiredState` gate); pubkey validated 44-b64/32-byte;
  **DELETE takes the pubkey in the JSON body** (base64 `/`+`+` keep pubkeys out of URL paths);
  mutation responses carry `sync: ok | deferred: <err> | disabled` — the DB is the source of
  truth, a failed push defers to the reconciler.
- **Wiring** (`cmd/hub/main.go`): `WG_ENDPOINT_SSH_{ADDR,USER,KEY_FILE,HOSTKEY}` env (key from
  the mounted `Secret/wg-endpoint-ssh`, host key non-secret plain env); any piece missing →
  `[INFO] WG peer-sync disabled` and mutations still work DB-only.
- **Tests:** allocator (exact IPs, freed-IP reuse, /30 exhaustion), API auth/validation with a
  fake syncer, SSH client against an in-process server (exact payload bytes, stderr surfacing,
  wrong-host-key refusal, multi-hostkey pin), reconciler (full-list, retry-on-tick, no-mutation
  drift push, removed-peer-absent negative). Four red-proofs run and reverted (allocator-ignores-
  rows, gate removal, InsecureIgnoreHostKey, delta-only push) — each failed its test.
- **Live-validated** end-to-end on the dev endpoint (`felhom-hetzner`, runbook
  `documentation/runbooks/offsite-endpoint.md`): add → `wg show` on the box; delete → gone
  (+404/403 paths); malformed payloads leave wg state byte-identical; endpoint reboot → persisted
  set + hub push converges; client tunnel `ep0.felhom.eu:443` → PBS login page via the wg0-only
  8007 rule; public 8007 unreachable. v0.32.1 = the HostKeyAlgorithms fix (0.32.0 image was
  already pulled by the cluster; tag kept immutable).

## docs — Felhom skills introduced + CLAUDE.md refresh (2026-07-03)

Repo-level docs work alongside v0.31.0 (no hub code in this entry):
- **`skills/`** (new, repo root): three versioned Claude Code skills — `felhom-build-deploy`
  (per-artifact runbooks, all commands verified live), `felhom-ui-design` (v2 tokens + gates),
  `felhom-testing` (non-hollow doctrine + red-proof procedure). Installed to `~/.claude/skills/`
  via `scripts/install_skills.py` (junction mode verified).
- **CLAUDE.md refresh:** the "Hub — current state (v0.7.x)" narrative (stale by ~23 versions)
  replaced with a version-free architecture section; standing rule adopted — CLAUDE.md carries NO
  version-pinned state (that lives in CONTEXT/CHANGELOG/REUSE); skills pointers added. Same rule
  applied to the sibling repos' CLAUDE.md in their own commits.

## v0.31.0 — critical severity accepted at event ingest + visible in UI (2026-07-03)

Fixes the gotcha the REUSE sweep surfaced: `handleEvent` coerced any severity outside
{info,warning,error} — including `"critical"` — to `"info"` at ingest, so a controller-POSTed
critical event never notified even though the dispatcher (`severityNotifies`, v0.24.0) and
`FormatOperatorEmail` already handle critical correctly.

- **Ingest (`internal/api/handler.go` `handleEvent`):** `"critical"` added to the severity case
  list. Unknown values (and case-variants like `"Critical"`) still coerce to `"info"` — the
  exact-match-lowercase coercion contract is kept and now locked by test.
- **Hungarian label (`internal/notify/templates.go`):** `severityLabels["critical"] = "Kritikus hiba"`
  (was missing — customer emails would have shown the raw English word).
- **UI counts:** dashboard consumer (`internal/web/server.go`) gains `EventCriticals`;
  `dashboard.html` renders the critical badge FIRST in the 24h count chain (guard extended);
  `customer_unified.html` gains the `{{.}} critical` summary badge before errors.
- **style.css:** defines the previously-referenced-but-undefined `.severity-critical`
  (`--crit`/`--crit-dim` tokens) and `.severity-ok` (neutral, exception-color principle). No other
  restyle.
- **Tests (`internal/api/event_test.go`, new):** critical preserved to store (companion red-proof:
  shown failing against the pre-fix switch — stored `"info"`); unknown severity → info; unknown
  event_type → 400 + nothing stored; no-auth → 401. First tests on the /event endpoint.
- REUSE.md §1/§3 updated in the same commit (the maintenance rule's first outing).

## docs — REUSE.md introduced (2026-07-03)

Cross-repo reuse-map rollout (docs-only, no code change, no version bump). New `REUSE.md` at the
repo root covering hub + website + scripts + manifests: canonical helpers (34 rows), patterns
(monitor checker, website page, gate script, GitOps deploy), dangerous lookalikes (legacy /notify
trio, severity-`critical` coercion at handleEvent ingest, inline stringData secrets, kubectl-apply
drift…), seams, extension points, and observed duplication (5 clusters, NOT fixed). New
`scripts/reuse_refs_check.py` machine-checks every cited path in all four repos' REUSE.md files.
CLAUDE.md gains the REUSE.md pointer + same-commit maintenance rule.

## v0.30.1 — status badge no-wrap (2026-07-02)

Found in the authenticated D4 validation pass: multi-word status tags (PENDING in a narrow
dashboard column, NO REPORT on hosts) wrapped between the CSS dot and the label. One line:
`white-space: nowrap` on `.status-badge`.


## v0.30.0 — TASK-D4: design system v2 re-skin (appearance only) (2026-07-02)

Last surface of the design sprint (controller D0/D1, website D3). The hub leaves its Tailwind-slate
theme for the canonical navy v2 language. **API surface untouched** (`/api/*` ingestion, artifact
manifest, config generation, DR/escrow — `git diff` clean under internal/api + internal/store).

- **Fonts (`internal/web/static/fonts/`, embed.go, server.go):** the 4 vendored woff2 (byte-copied
  from felhom-controller; latin-ext for Hungarian customer names in an English UI), embedded and
  served at `/static/fonts/` (font/woff2, immutable), mirroring the chart.min.js pattern. No CDN
  before or after.
- **`statusColor` semantic remap (server.go):** returns `nominal/warn/crit/neutral` class tokens
  instead of raw hex colors — ok→nominal, warn+stale→warn, down+fail→crit, pending+disabled→neutral
  (a not-yet-provisioned or deliberately paused customer is a normal fleet state), blocked→warn
  (intentional operator cut-off: attention-worthy, not an outage). The inline
  `style="color: {{statusColor}}"` pattern is dead (dashboard + customer_unified use class-based
  `.status-dot-<token>`); `statusIcon` ("●") retired. Truth-table test red-proven vs the old
  implementation; new template-parse test (neither existed for the hub).
- **style.css v2:** navy tokens + @font-face; 2px radius; hairline `--line-soft` table rows
  (fleet-NOC density kept); `.status-badge` re-expressed as an outline tag + CSS dot per the
  design-system addendum (ok=blue, warn/blocked/stale=amber+dim, down/fail=red+dim,
  pending/disabled=quiet neutral with hollow dot); severity badges stay filled amber/red
  (exceptions stay loud); config badges = filled informational chips in v2; row tint only for
  warn/down. Two-tone brand H1 (`Felhom <span>Hub</span>`) on all pages; 12-symbol Lucide sprite
  partial included per page.
- **Charts (app_detail):** avg memory `#2EA8F5`, peak `#8E7CE8` (secondary DATA series — not
  status red), catalog-limit line `#E0A93E` (threshold marker); legend/tick/grid → v2 literals.
- customer_unified JS status-message colors → blue-bright/crit; login page inline HTML retinted.
- Grep gate: all slate hexes (`#0f172a #1e293b #334155 #60a5fa #4ade80 #facc15 #f87171 #94a3b8
  #64748b #475569 #e2e8f0`) at zero across internal/web (non-test).

## v0.29.0 — Day-0 artifact manifest: version dropdowns + auto-derived sha (2026-07-01)

Removes the hand-copied sha256 from the Day-0 artifact manifest. The operator now **picks a version**
from a dropdown of what's actually in Gitea (olders get pruned), and the hub **reads that version's
sha256 from Gitea itself** — no transcription, no stale checksums. Keeps the human-in-the-loop trust
gate (the operator still deliberately chooses the version; "latest" is never auto-promoted) while the
hub stays the checksum trust root.

- **`internal/gitea`** (new): a minimal read-only Gitea packages client — `ListVersions` (generic
  package versions, newest-semver first) + `FileSHA256` (a version's file sha256 via the files-metadata
  API, **without downloading** the artifact — important for the ~GB golden). Basic-auth with the
  registry creds the hub already holds. Unit-tested against an httptest server (filter+sort, preferred
  file match + fallback, non-200 → error).
- **Configuration → Day-0 artifacts:** the two version text inputs are now `<select>` dropdowns
  populated from Gitea; the sha256 fields are **read-only, displayed** (mirrored from the picked
  version via a tiny inline script). Choosing "— none —" clears an artifact.
- **`handleSetArtifacts`:** derives each chosen version's sha256 from Gitea **authoritatively** (a
  client-submitted sha is ignored); a Gitea lookup failure REFUSES the save (never stores a version
  with a wrong/blank checksum) rather than corrupting the manifest.
- **Graceful degradation:** with no registry creds (`web.SetGiteaClient` not wired) the form falls back
  to the previous manual text-entry path. `main.go` enables the Gitea browser when `REGISTRY_USERNAME`
  /`REGISTRY_TOKEN` are set.
- `go build`/`vet`/`test ./...` clean.

## v0.28.0 — global settings → Configuration tab + online setup command (2026-07-01)

Three operator-requested improvements (companion: host-install script v1.2.0).

- **Global settings moved from the Customers page to the Configuration tab** (`web/configuration.html`,
  `configs.html`, `server.go`, `configs.go`). The two **global** cards — "Managed updates — global floor"
  and "Day-0 artifacts — agent & golden" — were on the Customers list; they now render + save on
  Configuration (where they belong). `handleConfiguration` supplies `GlobalFloor` + `Artifacts` +
  `CSRFField`; the save handlers (`handleSetGlobalFloor`/`handleSetArtifacts`) now redirect to
  `/configuration?flash=…` and are mounted at `/configuration/global-floor` + `/configuration/artifacts`;
  their flash banners moved too. The Customers page is back to just the list + "Add Customer" (its
  per-customer effective-floor column is unchanged).
- **Online setup command added** to a customer's Setup Command block (`customer_unified.html`). New
  **"Option 1: Online install (recommended)"** — download-then-run: `curl -fsSL
  https://felhom.eu/scripts/felhom-host-install.sh -o … && sudo bash … --customer-id <id>` — with a copy
  button and the customer id filled. The passphrase is **not** templated in (entered at the prompt). The
  former local-file command becomes Option 2, the debug curl Option 3. Download-then-run (not
  `curl | sudo bash`) stays the recommended form — inspect before running.
- **Website now serves `/scripts/`** (`manifests/webpage.yaml`). The host-install script lives at the
  repo's `/scripts` (outside the website doc-root); added `/scripts/` to the git-sync sparse-checkout and
  an nginx `location /scripts/` (root `.../current`, `text/plain`) so
  `https://felhom.eu/scripts/felhom-host-install.sh` resolves — single source of truth, no duplicated copy.
- **Remaining audit follow-ups unchanged:** controller-side geo intent sync; a read-only reported-vs-desired
  "Show Diff"; the cosmetic `controllerURL` cleanup in `configs.go`.

## v0.27.0 — Hosts page: read-only fleet view (audit F-M1) (2026-07-01)

Resolves audit finding **F-M1**: the agent enrolls as a *host* and the hub stores rich host state
(identity, agent version, guests, storage targets with SMART, DR/escrow, staleness) and alerts on it —
but the whole host domain was **invisible in the GUI** (email-only). Adds a **Hosts** nav section — a
fleet list + a per-host detail page. **Read-only** (GET only, no host actions/mutation routes): this
surfaces state the way the pull/desired-state model demands; it does not reintroduce inbound control
(retired in v0.26.0).

- **New store reader `ListGuestsForHost(hostID)` (`internal/store/store.go`).** `SELECT … FROM guests
  WHERE host_id = ? ORDER BY vmid`, via a new `scanGuest` helper over `guestRealitySelectCols` — the
  reality columns only. It deliberately **omits the secret/inert columns** (`api_key`,
  `desired_spec_json`), so the read-only view can never surface them. Returns `[]` (never nil-error) on
  no guests. (There was previously no guests *reader* — only `UpsertGuestFromReport`.)
- **New handlers (`internal/web/hosts.go`).**
  - `handleHostsList` — `ListHosts` + a per-host status badge from `hostStatus()` (which reuses
    `s.staleThreshold` — the **same** thresholds as the `HostStalenessChecker`: stale after the
    threshold, down at 2× — so the badge agrees with the alerting) + per-host guest counts
    (`ListGuestsForHost`) + vitals parsed from `GetLatestHostReportJSON` + worst storage fill grouped
    from `GetHostStorageTargets`.
  - `handleHostDetail` — `GetHost` (404 if absent) + `ListGuestsForHost` + rich storage targets (role,
    state, fill %, thin-pool, SMART health/temp/wear parsed from the latest report body) + vitals +
    `GetHostDRBundle`/`GetHostEscrow` **presence booleans only** (never the opaque blobs). Nil/missing
    (no report, no guests, no storage, no DR) render empty states — never a panic.
- **New templates `hosts.html` + `host_detail.html`** (existing dark operator-console styling reused —
  `data-table`, `status-badge-*`, `info-grid`, `empty-state`; no restyle). A no-report host shows a
  STALE/NO-REPORT badge and "waiting for first report".
- **Nav:** added the `Hosts` link (between Apps and Configuration) to every page's `<nav>` (the nav is
  duplicated per page, not a shared partial) + the new `timeAgoPtr` template helper for `*time.Time`.
- **Routes (`internal/web/server.go`):** `GET /hosts` (+ `/hosts/`) → list, `GET /hosts/{id}` → detail,
  modelled on the `/apps` pair. GET only.
- **Tests:** `ListGuestsForHost` (none→empty, multiple→vmid-ordered, secret column not surfaced); the
  list handler (N rows, ONLINE + NO-REPORT badges, worst-fill, no action buttons); the detail handler
  (guests + storage + SMART + DR present, customer cross-link, **no-secret assertion** that the host
  `api_key` is absent from the rendered body, no buttons); unknown host → 404; no-report host renders
  the waiting state. `hostStatus` band mapping unit-tested (pending/ok/stale/down).
- **Remaining audit follow-ups (not this slice):** controller-side geo intent sync; a read-only
  reported-vs-desired "Show Diff"; the cosmetic `controllerURL` cleanup in `configs.go`.

## v0.26.0 — pull-based config delivery + retire the inbound GUI controls (2026-06-30)

Closes audit `documentation/audits/AUDIT-hub-gui-2026-06-30.md` F-S1/F-S4 + the dead-template findings,
and replaces the never-inbound-violating "Push Config" with a pull-based config-refresh that rides the
report ACK (companion controller change: felhom-controller v0.94.0).

- **Config delivery is now pull-based (`internal/store/store.go`, `internal/api/handler.go`).** New
  `customer_configs.config_version` column — a **stored counter** (NOT a hash of the rendered YAML;
  `configgen` emits a fresh `web.session_secret` + timestamp every call, so a content hash would change
  spuriously). `SaveCustomerConfig` **bumps it on every save** (new rows seed at 1, updates increment) —
  the one path that changes the generated `controller.yaml` (identity + the `config_json` overrides). The
  floor, block/unblock, and retrieval-password regen deliberately do NOT bump it. The report ACK
  (`handleReport`) now advertises `config_version` beside `min_controller_version`/`latest_version`; the
  controller compares it to its last-applied version and re-pulls + self-restarts on a change. Omitted for
  report-only (no-config) customers, so an old controller is unaffected.
- **Retired the five inbound (hub→box) controls** that violated the never-inbound posture
  (`01-topology-and-trust.md:11`) and were broken behind the box's CF tunnel/NAT:
  - **Trigger Update** — handler + route deleted; controller updates are agent-driven (the version floor).
  - **Push Config** — handler + route deleted; replaced by the pull-based config-refresh above.
  - **Pull Config** — handler + route deleted.
  - **Show Diff** (`handleConfigDiff` + the `compareYAMLValues`/`flattenYAML`/`maskSensitive` helpers) —
    deleted, along with the now-dead `ConfigSyncStatus`/`ConfigDiffCount` plumbing.
  - **Geo-disable** — KEEPS its legitimate hub→Cloudflare WAF-rule removal (`RemoveGeoRules`); the
    secondary inbound `notifyControllerGeoDisable` is deleted. After this, `grep client.Do
    internal/web/` has **zero** ControllerURL targets (only Gitea registry/template fetches remain; the
    ControllerURL is still shown as a display-only link).
- **GUI staleness (F-S1) + dead templates:** the customer page's Setup Commands now show the Proxmox
  Day-0 host bootstrap (`sudo ./felhom-host-install.sh --customer-id <id>`, passphrase at the no-echo
  prompt) instead of the pre-Proxmox `docker-setup.sh`; Option 2 relabelled "Manual config fetch (debug
  only)". Deleted the orphaned `customer.html` + `config_detail.html` (rendered by nothing; `/configs/{id}`
  redirects to `/customers/{id}`).
- **Audit doc (deferred line):** the GUI audit `documentation/audits/AUDIT-hub-gui-2026-06-30.md` (committed
  `e51e03b`) is the grounding for the above; its F-S1/F-S4 + dead-template findings are now resolved. Open
  follow-ups noted there remain: the Hosts page (F-M1), controller-side geo intent sync, and Show-Diff
  could return later as a read-only-vs-reported view.
- Tests: store `config_version` bump (create=1, edits increment, per-customer independent) + the
  no-bump red-proof; ACK carries `config_version` and omits it for report-only customers + the no-bump
  red-proof. `go build/vet/test ./...` green.

## v0.25.0 — per-storage worst-fill alerting (StorageFillChecker) (2026-06-30)

Generalizes the host-root disk alert (v0.23.0) to ANY reported storage target — so a dedicated
dump/backup volume, data drive, lvmthin pool, or PBS datastore filling toward failure pages the operator
with the storage named, even when host root itself is fine.

- **`internal/monitor/storage_fill.go` (NEW) — `StorageFillChecker`.** A per-target mirror of
  `HostDiskChecker` on the same 60s sweep: born/persistent (already-breached `(host,target)` keys left
  UNSEEDED → first `Check` emits), escalation-only emit, recovery re-arm, the dispatcher's 1h cooldown.
  State is keyed per **(host, target)** so targets alert independently. Emits distinct
  `storage_fill_warning` / `storage_fill_critical` at the **natural `critical` severity** (exercises the
  v0.24.0 dispatcher fix with a second real caller). Default thresholds 90/95, hub-config overridable
  (`alerting.storage_fill_warn_percent` / `_crit_percent`), independent of the host-root thresholds.
- **Root excluded (no double-alert):** the host root-backed builtin (`Type=="local"`, or a target mounted
  at `/`) is skipped — `HostDiskChecker` owns root. So a root-backed vzdump dump is one alert (from
  host_disk), and storage_fill uniquely covers OFF-root storage.
- **`internal/store/store.go`:** `GetHostStorageTargets()` + `HostStorageTargetRow` — parses
  `report_json.storage_targets[]` of each host's latest report (percent = `used_fraction`×100); modeled on
  `GetHostDiskUsage`, no denorm column / migration.
- **`internal/notify/templates.go` + `internal/api/handler.go`:** Hungarian templates + allowlist entries
  for `storage_fill_warning` / `storage_fill_critical`.
- **`cmd/hub/main.go`:** register `storageFillChecker` on the 60s tick beside `HostDiskChecker`.
- Tests: per-target bands (independent warn/escalate/recover/re-arm), **born/persistent companion red-proof**
  (a seed-all model stays silent on the born-breach), **root-exclusion companion** (without the exclusion a
  root target IS in the critical band — the exclusion is what suppresses the double-alert), severity
  `critical`, and the store parse. `go build/vet/test ./...` green.

## v0.24.0 — dispatcher routes `critical` severity (+ nil-prefs crash guard) (2026-06-30)

NAS Part A2's "Part 0": close the dispatcher's silent drop of `critical`-severity events.

- **`internal/notify/dispatcher.go` `ProcessEvent`:** the severity gate was `severity != "warning" &&
  severity != "error"` → a `critical` event was **silently dropped** (never emailed). Now routes
  **warning / error / critical** (`severityNotifies`); `info` stays an intentional non-notify; any
  **unrecognized** severity is **logged** (`[WARN] Dispatcher: unrecognized severity …`), never silently
  dropped. Verified safe first: no controller event emits `critical` (all are info/warning/error) and the
  hub's only would-be `critical` emitter is `host_disk` — so no surprise alert volume.
- **`internal/monitor/host_disk.go`:** `host_disk_critical` now emits its **natural `critical` severity**
  (was forced to `error` to survive the old gate); `FormatOperatorEmail` styles `critical` 🔴 like `error`.
- **Latent crash guard:** `processCustomer` dereferenced `GetNotificationPrefs`, which returns `(nil, nil)`
  for a customer with no notification row — an event for such a customer would **panic the dispatcher
  goroutine and crash the hub**. Now guards `prefs == nil` before use.
- **Seam:** `sendEmailFn` field (defaults to the Resend sender) so routing is unit-tested without real HTTP.
- Tests: `severityNotifies` (warning/error/critical notify; info/unknown don't) + **companion red-proof**
  (the pre-fix `warning||error` predicate drops `critical`); ProcessEvent routes `critical` to the operator;
  an unknown severity is logged not dropped; `info` is silent and not mis-logged. `go build/vet/test ./...`
  green.

## v0.23.0 — host root-disk pressure monitoring + alert (2026-06-30)

Closes the silent-failure gap behind the felhom-pve incident: a Proxmox host root fs filling up (vzdump
piling under `/var/lib/vz/dump`) went unnoticed because nothing alerted on the HOST root `disk_percent` the
agent already reports. New hub-side checker on the existing 60s sweep.

- **`internal/monitor/host_disk.go` (NEW) — `HostDiskChecker`.** A sibling of `HostCapabilityChecker` /
  `HostLeafChecker`: reads each host's latest root-fs `disk_percent` (`store.GetHostDiskUsage`) and emits an
  operator alert on a warning (default **90%**) or critical (default **95%**) crossing. Rank-based bands
  (ok→warning→critical) so an escalation always alerts and a de-escalation/recovery re-arms silently.
  - **Born/persistent (the F2 lesson):** a disk ALREADY over threshold when the hub/checker (re)starts
    alerts on **cycle 1** — seeding leaves already-breached hosts UNSEEDED so the first `Check` emits (a
    transition-only design would stay silent forever on a persistently-full disk). The dispatcher's 1h
    operator cooldown dedups re-emits across a hub restart.
  - **Distinct event types** `host_disk_warning` / `host_disk_critical` — NOT the controller's GUEST
    `disk_warning`/`disk_critical` (the guest cgroup view), so the host and guest alerts never dedup or mask
    each other.
  - **Severity:** warning band → `warning`; **critical band → `error`** (NOT `"critical"`). The dispatcher
    only routes `warning`/`error` severities — a `"critical"` severity would be silently dropped — so the
    critical band maps to `error` (and the operator email's 🔴). (Deviation from the task's stated
    "critical → critical", made to match the live dispatcher.)
  - **Thresholds** are hub-config overridable (`alerting.host_disk_warn_percent` /
    `host_disk_crit_percent`, seed-only); an unset/invalid/misordered config falls back to 90/95
    (`normalizeDiskThresholds`) so a typo can never silence or invert the alert.
- **`internal/store/store.go`:** `GetHostDiskUsage()` + `HostDiskRow` — latest report per host (MAX(id)),
  `disk_percent` from the denorm column + total/used bytes parsed from `report_json` (event detail). No
  schema migration.
- **`internal/notify/templates.go`:** Hungarian customer templates for `host_disk_warning`/`_critical`
  (customer delivery still requires per-customer opt-in via enabled events; operator alert is the headline).
- **`internal/api/handler.go`:** `host_disk_warning`/`host_disk_critical` added to `allowedEventTypes`.
- **`cmd/hub/main.go`:** register `hostDiskChecker` on the shared 60s tick.
- Tests: band transitions (seed/escalate/steady/recover/re-arm), severity mapping, threshold defaults, and
  the **born/persistent companion red-proof** (a seed-all/transition-only model stays silent on a
  born-breach; the real unseeded design emits). `go build/vet/test ./...` green.
- **Follow-ups (noted, not built):** per-storage `StorageTargets` worst-fill alerting (a dedicated
  dump/backup storage filling — host root `disk_percent` already covers the observed case); and the
  provisioning-side `prune-backups` retention default so a box can't refill its own root (operational fix,
  separate from this detector).

## hub-config — enable operator email alerts (config-only, no image change) (2026-06-30)

`manifests/hub.yaml` `hub-config` ConfigMap: set `notifications.operator_email: admin@felhom.eu` +
`operator_enabled: true`. The dispatcher's operator path (`Dispatcher.processOperator`) sends only when
`operatorOn && operatorEmail != ""`; without these it returned early, so the self-health pipeline
(probe → report → checker → dispatch) stopped one hop short of the inbox (the TESTRUN's unproven hop).
**No hub image change** (live tag stays v0.22.1) — ConfigMap edit + pod restart to reload.
- **Proven end-to-end (2026-06-30):** a real capability-degrade alert produced
  `[INFO] Operator email sent for demo-felhom/agent_capability_degraded` (the send-success line that
  never fired while the path was gated); the customer path (`POST /api/v1/notify` `event_type:test`) sent
  to the customer address via the same `sendEmail` → Resend. See
  `documentation/audits/TESTRUN-fullstack-2026-06-29.md` ("Findings closed", Part A). Operator email is not
  a secret; the Resend key stays injected from `Secret/resend-api`.

## v0.22.1 — wire HostLeafChecker into the monitor loop (v0.22.0 missed the wiring) (2026-06-29)

The v0.22.0 commit added `HostLeafChecker` but the `cmd/hub/main.go` goroutine edit never applied, so
the checker was never started. The live test caught it (a leaf regen produced no `host_leaf_changed`,
only the controller's complementary `agent_channel_pin_mismatch`). Now started on the 60s sweep next to
the staleness/capability checkers. No other change.


## v0.22.0 — proactive agent re-key detection: HostLeafChecker (host_leaf_changed) (2026-06-29)

Companion to felhom-agent v0.48.0 (which now reports its served local-API leaf fp). The hub watches
each host's leaf fingerprint and raises an operator alert when it **changes** (an agent re-key) —
proactive, fleet-wide, independent of any controller's channel-health check. The last self-health leg.

- **`monitor.HostLeafChecker` (NEW):** sibling of `HostCapabilityChecker`. Trust-on-first-report — the
  first fp per host is the baseline; a later change emits **`host_leaf_changed`** (operator, English;
  details carry old+new fp) and advances the baseline. First-obs seeds silently (a "change" needs a
  prior value, so no F2 issue). An empty reported fp (pre-v0.48.0 / local-API-disabled) is unknown —
  never seeds, never alerts, never overwrites a baseline. Customer-blocked hosts dropped; unseen pruned.
  Runs on the existing 60s sweep. **KNOWN LIMITATION (documented):** trust-on-first-report can't detect
  a re-key that happened before the hub's first report — but the controller channel-check catches the
  downstream pin mismatch, so this is defense-in-depth.
- **`store.GetHostLeafFingerprints` (NEW):** latest reported fp per host, parsed from `report_json`
  (mirrors `GetHostCapabilities` — `MAX(id)`, **no schema migration**).
- **No allowlist change:** `host_leaf_changed` is hub-GENERATED (via `SaveEvent` + `dispatcher.ProcessEvent`),
  not controller-pushed, so it bypasses the `/api/v1/event` `allowedEventTypes` gate — same as the
  host_* events. The generic operator template relays it (no template change).
- Tests: change **red-proof** (A→B → one event + baseline advanced; companion: unchanged → none),
  first-obs seeds silently, change-back re-alerts, empty fp skipped, customer-blocked dropped. Cross-repo
  golden mirrors `leaf_fingerprint`. Version `0.21.0 → 0.22.0`.


## v0.21.0 — F2: alert on a host already degraded/stale at hub (re)start (2026-06-29)

Mirror of the controller's F2 fix, for the hub checkers: a host that was already **degraded**
(capability) or **stale/down** (staleness) when the hub (re)started was seeded silently and **never
alerted**. Now the constructors seed only HEALTHY hosts; an unhealthy host is left **unseeded** so the
first `Check()` emits once. The dispatcher's 1 h operator cooldown dedups the re-emit across a hub
restart (so a hub bounce doesn't re-page for an already-known issue within the window).

- `internal/monitor/host_capability.go` + `host_staleness.go`: constructor seeds only the healthy
  state; `Check()` first-obs (`oldState==""`) emits `emitTransition(…, "unknown", newState, …)` when
  the observed state isn't healthy, else seeds silently.
- Tests: born-degraded **red-proof** (host degraded at construction → unseeded → one
  `agent_capability_degraded` on first Check, no duplicate on the next); the staleness test updated to
  the F2 behavior (born-stale → unseeded → one `host_stale` on first Check). Version `0.20.0 → 0.21.0`.


## v0.20.0 — Accept controller `agent_channel_*` events (channel-health relay) (2026-06-29)

Companion to felhom-controller v0.90.0's controller→agent channel health-check. The controller pushes
classified `agent_channel_*` events to `/api/v1/event`, but the handler's `allowedEventTypes`
allowlist rejected them (HTTP 400). Added the 8 types (`pin_mismatch`, `unauthorized`, `unreachable`,
`timeout`, `misconfigured`, `construction_error`, `unknown`, `recovered`) so the operator relay works.
These are **operator-only** (not customer notification toggles, same as the host_* events); the
existing dispatcher routes them generically (no template change). Version `0.19.0 → 0.20.0`.

## v0.19.0 — Agent capability-degraded operator alert (HostCapabilityChecker) (2026-06-29)

Companion to felhom-agent v0.44.0's privileged-capability self-probe: the agent now rides a
`capabilities` snapshot on its host report (each required `sudo -n` grant: ok/degraded), and the hub
alerts the operator when a host transitions into a degraded state — closing the loop that let five
non-root-cutover regressions go undetected until user-visible breakage.

- **`monitor.HostCapabilityChecker` (NEW):** a deliberate SIBLING of `HostStalenessChecker` — same
  per-host state map (`ok`/`degraded`), seed-without-event, emit-only-on-transition shape. A host is
  `degraded` iff its latest report has any **Critical** capability with `status:"degraded"`;
  non-critical degradations ride the report but never alert. Runs on the existing 60s sweep next to
  the staleness checkers.
- **Events:** `agent_capability_degraded` (warning) on ok→degraded, naming the degraded capabilities
  + gated features in the message and details JSON; `agent_capability_recovered` (info) on
  degraded→ok. Routed through the existing `Dispatcher.ProcessEvent` — **operator-only** (the type
  is not a customer notification toggle, same as `host_stale`) with the standard 1 h operator
  cooldown (no per-cycle re-alert).
- **`store.GetHostCapabilities` (NEW):** reads the capability snapshot from the latest host-report's
  `report_json` per host (keyed on `MAX(id)` — within-second `received_at` ties would otherwise
  return multiple rows). **No schema migration** — the array rides the existing report body. A
  pre-v0.44.0 agent (no `capabilities`) reads as `ok`, so an old agent can't trip a false alert.
- Cross-repo `host-report.golden.json` mirrors the new `capabilities: []` field (byte-identical with
  the agent copy). Version `0.18.0 → 0.19.0`.

## v0.18.0 — App-email passthrough: POST /api/v1/mail → Resend SMTP (2026-06-29)

The hub can now relay a customer box's outbound app email to Resend, re-emitting the raw MIME
**unchanged** over SMTP. This is the hub leg of the app-email relay (apps → on-box shim → hub →
Resend); the Resend key stays hub-side. Implements `documentation/audits/SPIKE-smtp-app-relay-2026-06-28.md`.

- **New `internal/mailrelay/relay.go`:** `ResendSMTP` (a `Sender`) — STARTTLS to `smtp.resend.com:587`,
  `AUTH LOGIN resend/<key>` (a small `net/smtp.Auth` LOGIN impl; stdlib ships PlainAuth only), then raw
  `MAIL`/`RCPT`/`DATA`. **Raw passthrough** — NOT the `internal/notify` Resend **HTTP-API** path, which is
  unchanged for the hub's own structured alerts and silently drops inline CID images (spike §4). No new
  external dependency (stdlib `net/smtp`).
- **New `POST /api/v1/mail`** (`internal/api/mail.go`): authenticates the box (`checkAuthCustomer`), enforces
  the **From-header** domain allowlist (backstop; reject 403), applies a **per-customer in-memory token-bucket
  rate limit** (default 30/min → 429 so one box can't drain the shared Resend quota), then passes the raw
  bytes through to Resend. Success→200, Resend failure→502 (the box's shim maps that to the app).
- **Config:** new `mail` section (`per_customer_per_minute`, `from_domains`); wired in `cmd/hub/main.go` only
  when a Resend key is present (else the endpoint returns 503). The existing `notify/dispatcher.go` alert path
  is untouched.
- **Tests:** passthrough byte-equality (raw bytes reach the sender unchanged, not parsed), From-reject +
  companion red-proof, per-customer rate-limit + isolation + companion, send-failure→502, 401/503/400 paths,
  token-bucket unit (injected clock), LOGIN auth + From-domain parsing.

## v0.17.0 — Resend key sourced from a Secret, out of git (2026-06-29)

Resend rotation + de-git hygiene. The hub's Resend API key was committed in plaintext in
`manifests/hub.yaml`'s `hub-config` ConfigMap; it is now sourced from an out-of-band Kubernetes
Secret (`resend-api`) and never lives in git.

- **`cmd/hub/main.go`:** new `RESEND_API_KEY` env override for `notifications.resend_api_key`, mirroring
  the existing `REGISTRY_TOKEN` / `DEFAULT_MIN_CONTROLLER_VERSION` k8s-Secret override pattern. When set,
  it wins over the (now empty) ConfigMap field. No behaviour change when unset.
- The committed ConfigMap `resend_api_key` is now an empty placeholder pointing at
  `documentation/runbooks/secrets.md`; the live value is injected from `Secret/resend-api` via env.
- Part of the Resend key rotation: the previously-exposed send-scoped key was rotated; the new key lives
  only in the out-of-band store and `Secret/resend-api` (created imperatively, never committed). See the
  secrets runbook. No secret value appears in this repo.

## v0.16.0 — Day-0 artifact manifest (hub-vouched agent + golden checksums) (2026-06-28)

The hub now serves a passphrase-authed **artifact manifest** so the host-bootstrap script can
**fetch-then-verify** the agent binary + golden archive from Gitea before installing them. The hub is
the checksum **trust root** — a different root than Gitea (which only stores the bytes). Part of the
BUNDLE slice that lets a fresh PVE box self-install the agent (no more manual binary/unit step).

- **`store.go`:** `ArtifactManifest{agent_version, agent_sha256, golden_version, golden_sha256}` with
  `Get/SetArtifactManifest`, persisted as four discrete rows in the existing `hub_settings` key/value
  table (no schema change — same mechanism as the controller-version floor; survives restarts; partial
  sets round-trip). Added generic `getSetting/setSetting` helpers.
- **`handler.go`:** new `GET /api/v1/artifacts/{customer_id}` — auth mirrors `handleConfigRetrieve`
  EXACTLY (`X-Retrieval-Password`, 404-then-401 order, constant-time compare). Returns
  `{"agent":{version,sha256},"golden":{version,sha256}}`. An unset manifest returns **200 with empty
  fields** (not an error) so the script falls back to the local golden / fails clearly on a missing
  binary. v0.16.0 returns the GLOBAL current set for every customer (per-customer pinning is a future
  hook).
- **`configs.go` + `configs.html` (operator UI):** a "Day-0 artifacts — agent & golden" card beside the
  managed-update floor controls, with version + sha256 fields for each artifact. `POST /configs/artifacts`
  (CSRF-protected); versions validated as bare semver (reusing the floor validator), sha256s as 64-hex,
  blank-to-clear.
- **`main.go` (env seed):** seeds the manifest from `ARTIFACT_AGENT_VERSION` / `ARTIFACT_AGENT_SHA256` /
  `ARTIFACT_GOLDEN_VERSION` / `ARTIFACT_GOLDEN_SHA256` on startup — but only fields the DB doesn't already
  have, so a UI edit sticks across restarts. Same escape hatch the Phase-2 floor uses (the operator UI is
  password-gated).
- **Tests (`artifact_test.go`):** recorded set returned verbatim; unset → 200 empty; wrong/missing
  passphrase → 401; unknown customer → 404; store partial-set round-trip. Plus the floor-render smoke
  test updated for the new list-page data field.
- Reuse: no new auth path (passphrase, like config-retrieve + host-enroll); no new table (hub_settings);
  no new fetch credential downstream (the script reuses the config-retrieve git token).

## v0.15.0 — Phase 2 managed updates: per-customer controller-version floor (2026-06-27)

The operator can now set a **minimum controller version** (FLOOR) — per-customer, defaulting to a
**global** floor — and any box below it **auto-updates** to the floor on its next report (no customer
action). The customer "update to latest" button is unchanged (latest, opt-in); the floor is the
operator's enforced minimum and the **auto-target** (controlled rollout: the operator raises the floor).

- **`store.go`:**
  - `customer_configs` gains `min_controller_version TEXT NOT NULL DEFAULT ''` (per-customer override).
  - New `hub_settings(key,value)` table holds the operator-set **global** floor (survives restarts).
  - Global default fallback `SetDefaultMinControllerVersion` (config/env `DEFAULT_MIN_CONTROLLER_VERSION`).
  - `EffectiveMinControllerVersion(customerID)` = per-customer override if non-empty, else global
    (hub_settings → config default), else "". Plus `Set/GetGlobalMinControllerVersion`,
    `SetMinControllerVersion`.
- **`handler.go` (`handleReport`):** the controller report ACK (previously just `{status, customer_blocked}`)
  now also returns `{min_controller_version: <effective floor>, latest_version: <registry latest>}`. Both
  omitted when empty, so old controllers / unconfigured hubs behave exactly as before. New
  `SetLatestVersionProvider` wires the registry `VersionChecker` (nil-safe).
- **`configs.go` + templates (operator UI, English):** the Customers list shows a **Floor** column
  (effective floor, "override" tag, below-floor ● marker) + a **global floor** editor; each customer page
  shows the effective/global floor, below-floor status, and a **per-customer override** form. Routes
  `POST /configs/global-floor` and `POST /customers/{id}/floor` (CSRF-protected; X.Y.Z or blank-to-clear).
- **`main.go`:** seeds the default floor from config/env; wires the latest-version provider.
- **Tests:** effective-floor resolution (override beats global; empty when unset; DB-beats-default;
  preserved across save); report ACK carries effective floor + latest + omits when unset; template render
  smoke. **Companion red-proof:** breaking `EffectiveMinControllerVersion` to return the global when an
  override is set makes the override-precedence store test AND the ACK test FAIL (verified, then restored).
- Reuse: rides the existing report cycle (no new endpoint); the swap itself is the controller's Phase 1
  flow + the agent — untouched here.

## v0.14.0 — Passphrase-authed host enrollment (Day-0 option C) (2026-06-26)

Adds the single-secret Day-0 host-enrollment path proven in
`documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md` (option C). The operator /
host-bootstrap script now carries **only the customer's retrieval passphrase** — the global operator
key never enters the field deploy path.

- **New endpoint `POST /api/v1/host-enroll`** (`internal/api/handler.go`, `handleHostEnroll`):
  passphrase-authed (`X-Retrieval-Password` header, body `{customer_id}`), returns
  `{host_id, api_key}`. **Mint-once-reuse** — mints on first call (`201`), returns the existing
  credential byte-for-byte on every subsequent call (`200`), so re-running the bootstrap never orphans
  a running agent's key. Auth is checked **before** any mint (a wrong passphrase never writes a row):
  wrong/missing passphrase → `401`, unknown customer → `404`, missing `customer_id` → `400`. Mirrors
  `handleConfigRetrieve`'s auth pattern + `handleAdminCreateHost`'s mint block.
- **New store method `Store.GetHostByCustomer`** (`internal/store/store.go`): `SELECT … FROM hosts
  WHERE customer_id = ? ORDER BY updated_at DESC LIMIT 1` (uses `idx_hosts_customer`), nil-on-not-found.
  Backs the reuse lookup. >1 host for a customer (not expected in Day-0) → most-recent wins, never a
  duplicate mint.
- **Unchanged & deliberately untouched:** `GET /api/v1/config/{id}` (controller pull — same raw-YAML
  body) and `POST /api/v1/admin/hosts` (global-key operator escape hatch, still PROVISIONAL pending the
  cutover lock-down).
- Tests: `internal/api/host_enroll_test.go` (mint/reuse/401-no-mint/404/400, +5) and a
  `GetHostByCustomer` store test (+1); companion red-proof confirmed an always-mint variant fails the
  reuse assertion.

## v0.13.1 — DR recipe v1 drive-shape sync: test-data + regression guard only (2026-06-16)

**No behavior change — redeploy optional.** Tracks the agent's v0.39.0 v1 host-half drive shape (which
dropped `role` + `restic_repo_coord` from `drives[]`). Because the hub reads drives as `json.RawMessage`
(verbatim passthrough), no store/handler/struct change was needed — only test-data + a regression guard.

- `internal/store/testdata/dr-recipe.golden.json` + the `drHostHalf` test fixture: dropped the `role`
  key from `drives[0]` to match the v1 shape the agent now emits.
- `internal/api/testdata/host-report.golden.json`: re-synced to be **byte-identical** with the agent's
  `internal/hub/testdata/host-report.golden.json` (sha256 `57f2a5e7…18b2f2b5`). The hub copy previously
  lacked the `dr_recipe` section entirely; it is now a verbatim copy, so the cross-repo golden truly
  matches and POSTing it through `/host-report` exercises the `SaveDRRecipeHostHalf` ingest path.
- New `TestAssembleDRRecipe_V1DriveShape` — the regression guard: a stored host half whose `drives[]`
  carry NEITHER dropped field but WHICH HAS a `pbs` block assembles cleanly (pbs carried through, drives
  passed through verbatim, neither `role` nor `restic_repo_coord` present). Demonstrated to FAIL when the
  fixture re-adds `role`, then reverted.

## v0.13.0 — DR recipe: assemble + store + view the secret-free reconstruction recipe (2026-06-16)

**DR recipe slice (hub half)** — the assemble-store-view side of the secret-free reconstruction recipe
(`documentation/audits/SPIKE-dr-recipe-2026-06-16.md`). The hub receives two additive halves via the
existing report paths — the agent's storage/guest/PBS half (on the host-report) and the controller's
customer/apps half (on the controller report) — and assembles them into one operator-readable recipe per
customer. This is the clean inverse of the retired infra-backup: same "re-provision plan" goal, but
PLAINTEXT-at-rest is *correct* because the recipe has zero secrets.

- **Store** (`internal/store/dr_recipe.go`): a DEDICATED `dr_recipe` table (NOT `host_escrow`, NOT the
  dropped `infra_backup*` tables) keyed by `customer_id`, holding `host_half_json` + `app_half_json` +
  `recipe_version` + `host_id`. `SaveDRRecipeHostHalf` / `SaveDRRecipeAppHalf` each upsert their half and
  PRESERVE the other (last-write-wins per half). `AssembleDRRecipe` stitches the two into an
  `AssembledRecipe{recipe_version, customer, guests, pbs, drives, pve_storage, apps}` — sub-sections pass
  through as `json.RawMessage` (verbatim), **ignore-unknown** at the top level and version-skew tolerant
  (`recipe_version` = max of the two halves) for forward-compat across the three repos.
- **Ingest** (`internal/api/handler.go`): `handleHostReport` persists the `dr_recipe` host-half (keyed by
  the host's customer); `handleReport` persists the `dr_recipe` app-half (keyed by `customer_id`) —
  mirroring the `app_telemetry` pattern, backward-compatible (old agents/controllers omit the field), and
  never fatal to the heartbeat.
- **View** (`internal/web/dr_recipe.go` + customer page): a DR-recipe panel on the customer detail page
  (which half has landed + last-updated) with a **Download recipe (JSON)** link →
  `GET /customers/{id}/dr-recipe.json` serves the assembled recipe (operator dashboard-auth, pretty JSON,
  `Content-Disposition` attachment). No decrypt, nothing to redact.
- Tests: `TestDRRecipe_StoreRoundTrip` (each half preserves the other), `TestAssembleDRRecipe_MatchesGolden`
  (the assembled wire shape pinned in `testdata/dr-recipe.golden.json`),
  `TestAssembleDRRecipe_IgnoreUnknownAndVersionSkew` (a forward-compat half still assembles; version = max),
  `TestAssembleDRRecipe_PartialHalves` (one half present), `TestAssembleDRRecipe_NoSecrets` (defense-in-depth
  credential-key sweep). Pairs with felhom-agent v0.38.0 + felhom-controller v0.73.0.

## v0.12.0 — retire Infra Backup + purge its plaintext secrets + fix the daily backup-deadline email (2026-06-16)

Phase-1 of the Infra Backup retirement (per `documentation/audits/SPIKE-infra-backup-2026-06-15.md`).
The mechanism had been dead since slice 8C, yet the hub still stored each version as a **plaintext
JSON blob at rest** containing the customer's app-secret encryption key, restic password, and
Cloudflare tokens — a zero-knowledge violation. Its absence was also the root cause of the daily
`expected_backup_missed` false-alarm email.

### Changed
- **Backup-deadline check repointed to PBS freshness.** `monitor.CheckBackupDeadlines` no longer
  looks for a `backup_completed` event (no component emits it anymore — the disk-tier backup moved to
  the agent in slice 8C, so the check fired daily for every healthy customer). It now reads the
  customer's **latest agent host-report** and raises `expected_backup_missed` only on positive
  evidence: no PBS snapshot / successful vzdump at all, the newest backup older than **26h**, or the
  newest PBS snapshot's `verify_state == "failed"`. A fresh-but-not-yet-verified snapshot is **not** a
  failure (PBS verifies on its own cadence) — alarming on it would just re-create the false alarm. The
  **db-dump half is unchanged** (the in-guest controller still emits `db_dump_completed`). A customer
  with **no host-report** (legacy/defunct) gets no backup alarm here — liveness is the
  host-staleness checker's job. New store accessor `GetLatestHostReportJSON`. Tests:
  `internal/monitor/deadline_test.go` (fresh+verified→quiet, stale→alarm, failed-verify→alarm,
  no-report→quiet, db-dump half preserved, plus a pure `assessBackupFreshness` table). The
  fresh+verified→quiet test is the **companion**: it fails against the old event-based check.

### Removed
- **The Infra Backup feature**: ingest endpoint `POST /api/v1/infra-backup`, getters
  `GET /api/v1/infra-backup/{id}[/versions]` and their handlers; store methods
  `SaveInfraBackup` / `GetInfraBackup` / `GetInfraBackupByID` / `GetInfraBackupMeta` /
  `ListInfraBackupVersions` / `pruneInfraBackups` + the `InfraBackupMeta` / `InfraBackupVersion`
  types; the operator **"Infra Backup" panel** (`customer_unified.html`, `customer.html`). The
  `GET /api/v1/recovery/{id}` endpoint is kept but now returns **only** the generated `config_yaml`
  (no infra-backup payload). The customer-page **config-drift badge** that diffed against the stored
  controller.yaml is hidden (its at-rest source is gone); the live **"Show Diff"** path is unaffected.

### Security / migration
- **Plaintext secret purge.** `migrate()` now `DROP`s `infra_backup_versions` + `infra_backups` and
  runs **`VACUUM`** (+ `wal_checkpoint(TRUNCATE)`) so the freed pages holding the plaintext keys/
  tokens are **physically reclaimed**, not merely delinked. Gated on table existence so normal
  restarts don't pay the VACUUM cost.
- **Out of scope (flagged for the operator):** the exposed Cloudflare / hub / session credentials in
  the dropped blobs remain valid until rotated (operator step). Separately, the legacy `reports`
  table holds thousands of historical rows with a plaintext `restic_password` value from old
  controller versions — a distinct leak, not purged here (the live controller no longer sends it).

## v0.11.0 — slice 10D: DR capstone — recovery mode + re-enroll + directive serving (2026-06-10)

The hub half of the slice-10 DR capstone (closes slice 10). The hub ORCHESTRATES recovery but holds
**no usable secret and no Cloudflare write-power**: the escrow blobs it serves are opaque (need `R`,
which the hub never has), and the destructive tunnel/PBS rotation is the **operator's** step from a
trusted environment. A compromised hub can at most hand out opaque blobs + rotate/revoke its own
per-host credential — it cannot hijack a customer's tunnel.

### Added
- **`PUT /admin/hosts/{id}/recovery-mode`** (global key) — arm recovery mode with a bounded TTL
  (`ttl_seconds`, clamped [60s, 4h], default 30m → **auto-expires**); **`DELETE`** to disable. The
  restore directive + re-enroll are served ONLY while recovery mode is active.
- **`POST /hosts/{id}/re-enroll`** — gated ONLY on recovery mode (the lost box has no old key; the
  operator armed recovery mode after out-of-band validation). Rotates the host's API key to the new
  box's key (**the old box's hub access is revoked instantly**) and returns the DR directive + the two
  **opaque** escrow blobs. Without recovery mode → 403. Zero-knowledge: even a wrongful re-enroll in
  the window leaks nothing recoverable (the blobs need `R`).
- **`GET /hosts/{id}/restore-directive`** (re-enrolled key, recovery-gated) — re-fetch the directive.
- **Store/escrow**: `hosts.recovery_mode_until` (additive); `host_escrow.identity_blob` +
  `directive_json` (the age-wrapped identity blob + non-secret directive, stored alongside the
  K-escrow). Methods: `SetRecoveryMode`/`ClearRecoveryMode`, `RotateHostAPIKey`, `SaveHostDRBundle`/
  `GetHostDRBundle`. The slice-7 escrow upload (`PUT /hosts/{id}/escrow`) now also accepts
  `identity_blob_b64` + `directive` (additive).

### Not built (by design — the locked rotation model)
- **No Cloudflare write-credential in the hub.** The operator deletes the stale tunnel connector +
  rotates the tunnel/PBS token from their trusted environment (a documented procedure / future small
  operator CLI). The hub may optionally hold a read-only CF token to surface connector state.

### Tests
- re-enroll refused without recovery mode (403); recovery-mode arm is global-key-only; re-enroll
  **rotates + revokes** (old key → 401, new key → 200); directive served only in recovery mode +
  **expires**; clear disables re-enroll.

## v0.10.0 — slice 10B: signed-op job completion (clear-job) (2026-06-10)

The hub half of slice 10B is small by design — the hub stores + serves the operator-signed blobs
**opaquely** (it holds no signing key and can neither forge nor open them; the agent verifies +
executes). 10B adds the missing **completion** path so a processed job leaves the queue.

### Added
- **`DELETE /api/v1/hosts/{host_id}/jobs/{job_id}`** (per-host key, **self-scoped**; the global key
  may clear any) — the agent calls it after executing OR terminally rejecting a job. Idempotent
  (clearing an absent job is a clean 200). Store: `DeleteSignedJob`.

### Unchanged (already in 10A, reused by 10B)
- `POST /admin/hosts/{id}/jobs` (operator enqueues the signed blob), `GET /hosts/{id}/jobs` (the
  agent fetches), and the `has_signed_ops` envelope flag. The signed blob stays opaque on the wire
  (a base64 `{op_blob_b64, sig_armored}` envelope the agent parses) — **no jobs-wire golden change**.

### Tests
- `DELETE …/jobs/{id}` is self-scoped (host A cannot clear host B's job → 403) and idempotent.

## v0.9.0 — slice 10A: desired-state serving + signed-jobs queue (the "Down" channel) (2026-06-10)

The hub half of slice 10A: the hub now **serves operator intent** down to already-authenticated
hosts. The control envelope (the host-report response) stops returning placeholder
`desired_generation:0 / has_signed_ops:false` and carries the host's **real** generation + a
signed-jobs flag — the cheap change-notification the agent (v0.15.0) acts on. The heavy
desired-state moves only on a dedicated, self-scoped fetch.

### Added
- **`PUT /api/v1/admin/hosts/{host_id}/desired-state`** (global/operator key only) — sets a host's
  desired-state and **atomically bumps `desired_generation`**. The body is JSON the hub stores +
  serves **opaquely** (it validates only that it is well-formed JSON; the agent/CLI owns the
  schema). Unknown host → 404; malformed JSON → 400. Minimal admin path; rich editing UX is later.
- **`GET /api/v1/hosts/{host_id}/desired-state`** (per-host key, **self-scoped** — a host reads
  only its own; the global key may read any) — returns `{generation, desired_state}`. The agent
  fetches it when the envelope's generation advances past its cache.
- **`GET /api/v1/hosts/{host_id}/jobs`** (per-host key, self-scoped) — serves the host's pending
  **opaque** signed-op blobs (oldest first). The hub never forged, opened, or executes them
  (verify + run is slice 10B; this only serves the queue).
- **`POST /api/v1/admin/hosts/{host_id}/jobs`** (global key only) — enqueues a pre-signed opaque
  job blob. The minimal operator path to seed the queue; the hub holds no signing key.
- **Store**: a new `signed_jobs` table (per-host opaque blob queue); `SetHostDesired` (set + bump
  generation, atomic), `EnqueueSignedJob` / `GetSignedJobs` / `CountSignedJobs`. The `hosts` table's
  previously-inert `desired_json` / `desired_generation` columns are now live.

### Changed
- The host-report **control envelope** now reports the host's actual `desired_generation` and
  `has_signed_ops` (queue non-empty), both degrading safely to their old defaults on a store error
  (a heartbeat never fails on the control channel). `poll_interval_seconds` / `blocked` unchanged.

### Tests
- admin-set bumps the generation each write + the served state reflects the latest body; admin-set
  is global-key-only (per-host → 403, malformed → 400, unknown host → 404).
- `GET /desired-state` is **self-scoped** (host A's key → host B → 403; global → any; no token → 401).
- the envelope carries the current generation + `has_signed_ops` flips on enqueue; `GET /jobs` is
  self-scoped + serves the blobs oldest-first; admin enqueue is global-key-only.
- cross-repo golden round-trip: `testdata/desired-state.golden.json` set → fetched back unchanged
  (the opaque pass-through), **byte-identical** with felhom-agent's copy.

## (no version bump) — slice 9 cross-repo wire-contract: `host.cpu_temp_c` (2026-06-10)

Slice 9 adds a nullable **`cpu_temp_c`** field to the shared `HostMetrics` wire struct (the agent's
new CPU/chassis-temperature collector). The agent's host-report carries it too, so the hub's
**cross-repo host-report golden** (`internal/api/testdata/host-report.golden.json`) was updated to
stay **byte-identical** with `felhom-agent/internal/hub/testdata/host-report.golden.json` (the
duplicated-contract discipline; manual diff confirmed identical). **No hub code change** — the full
report_json already persists the field verbatim, and the hub does not surface CPU temp on the
operator dashboard yet (an optional later freebie). The golden-contract test (`host_test.go`) still
passes (the host parse-struct ignores the extra key).

## v0.8.0 — opaque PBS recovery-code escrow storage (slice 7, doc 03 §8a) (2026-06-10)

Hub half of slice-7 close-out: store the agent's **opaque** `R`-wrapped PBS-key escrow blob. The
default posture is zero-knowledge — the hub holds ciphertext it **cannot open** (it has no recovery
code; there is no decrypt path). Pairs with felhom-agent v0.9.0 (escrow creation). Consumption /
restore-mode serving is slice 10.

### Added
- **`PUT /api/v1/hosts/{host_id}/escrow`** — authed with the **per-host key** (a host may only write
  its own escrow; the global operator key is also accepted). Body mirrors the agent's emit struct
  (`blob_b64`, `key_fingerprint`, `posture`, `created_at`). Stores the decoded **opaque bytes
  verbatim**; rotation is last-write-wins. No serving this slice.
- **`host_escrow`** table (`host_id` PK, `blob` BLOB, fingerprint/posture/created_at). Store methods
  `SaveHostEscrow` / `GetHostEscrow` (`HostEscrow`). The hub never transforms or decrypts the blob.

### Tests
- Stores the opaque blob **verbatim** (round-trips byte-identical); rotation last-write-wins;
  rejects an absent/wrong key (401) and a host writing another host's escrow (403); bad/empty
  base64 → 400; the wire-contract key-set matches the agent's emit struct.

### Security note
The hub stores ciphertext only — holding the blob does NOT let Felhom read customer data
(separation principle, doc 03 §8a). The per-host-key gate scopes writes to the owning host.

## v0.7.5 — restore-test "passed with warnings" visibility (2026-06-09)

Hub half of `TASK — Restore-test must not false-fail on benign start warnings` (Phase B). The
agent (v0.7.0) now treats a guest-start advisory like the systemd-nesting warning as a PASS
(verdict is liveness, not the start exitstatus) and carries the warning text on the wire. This
makes that visible to the operator instead of indistinguishable from a clean pass.

### Added
- `hostRestoreTest.warnings` (`[]string`) + `warnings_recognized` (`bool`) mirror fields, matching
  the agent's `hub.RestoreTest` wire contract (`omitempty`; an absent `warnings_recognized` ⇒
  `false` ⇒ treated as the louder unrecognized case — a missing flag can only over-notice).

### Changed
- Host-report ingest now surfaces a **passed** restore-test that carried warnings:
  `[INFO] restore-test passed WITH WARNINGS (recognized)` when every warning is the known-benign
  anchor, escalated to `[WARN] … UNRECOGNIZED WARNINGS` otherwise — as loud as a failed PBS
  verify, so a real restore warning can't hide behind a green pass. A FAILED restore-test still
  logs the existing `[WARN] … FAILED`.

### Tests / contract
- `restore_tests[0]` in the host-report golden gains `warnings` + `warnings_recognized`; the golden
  stays **byte-identical** with felhom-agent's copy (sha256-verified) and the bidirectional
  key-set contract test now round-trips the new keys through `hostRestoreTest`.

### Not in this slice
- No dashboard widget: the hub web layer renders only controller-report data — there is no
  host-domain dashboard surface yet (guests/storage/restore_tests/pbs_snapshots are log+persist
  only, same as the failed-PBS-verify signal). Distinct dashboard treatment lands when the
  host-domain dashboard does (slice 10). The operator signal this slice is the log line.

## v0.7.4 — ingest agent pbs_snapshots (slice 6 Phase B) (2026-06-09)

The agent's slice-6 Phase B work populates the host-report's `pbs_snapshots` (the PBS offsite
inventory + per-snapshot verify-state). This is the hub half: accept + persist them. Minimal —
the rich offsite policy is hub-owned (slice 10); this mirrors what the agent reports.

### Added
- **`hostPBSSnapshot`** mirror struct in `hostReportPayload` (`internal/api/handler.go`) —
  field-for-field with the agent's `hub.PBSSnapshot` wire contract (namespace/backup_type/
  backup_id/backup_time/size_bytes/owner/protected/encrypted/verify_state/verify_upid).
  Persisted via `report_json` (no new columns — the slice-5/6A precedent).
- **A FAILED PBS verify is logged prominently** (`[WARN]` — the loudest offsite-DR signal,
  same treatment as a failed restore-test). The `host-report` info line now counts pbs-snapshots.
- **`testdata/host-report.golden.json`** updated with a populated `pbs_snapshots[0]`, kept
  **byte-identical** with felhom-agent's copy.
- **`TestHostPBSSnapshot_GoldenContract`** — the hub half of the bidirectional key-set test.

### Notes
- Backward-compatible: an agent that omits/empties `pbs_snapshots` is accepted unchanged.

## v0.7.3 — ingest agent backups + restore_tests (slice 6 Phase A) (2026-06-09)

The agent's slice-6 work populates the host-report's `backups` + `restore_tests` (the
self-restore-test result). This is the hub half: accept + persist them. Minimal — the rich
backup policy (schedule/retention/target selection) is hub-manifest-owned and lands at
slice 10; this slice only mirrors what the agent reports.

### Added
- **`hostBackup` / `hostRestoreTest`** mirror structs in `hostReportPayload`
  (`internal/api/handler.go`) — field-for-field with the agent's `hub.Backup` /
  `hub.RestoreTest` wire contract. Persisted verbatim in `report_json` (no new columns —
  slice-5 precedent).
- **A FAILED restore-test is logged prominently** (`[WARN]`, the loudest DR signal there is);
  a failed backup is logged too. The `host-report` info line now counts backups + restore-tests.
- **`testdata/host-report.golden.json`** updated with a populated `backups[0]` /
  `restore_tests[0]`, kept **byte-identical** with felhom-agent's copy.
- **`TestHostBackup_GoldenContract` / `TestHostRestoreTest_GoldenContract`** — the hub half of
  the bidirectional key-set test (round-trip the golden through the mirror, assert exact keys).

### Notes
- Backward-compatible: an agent that omits/empties these is accepted unchanged. The legacy
  controller report path is untouched (frozen until slice 10).

## v0.7.2 — ingest agent storage_targets (slice 5 Phase A) (2026-06-09)

The agent's slice-5 work populates the host-report's `storage_targets` (previously empty).
This is the hub half: accept + persist them. Minimal by design — the rich, authoritative
storage manifest (desired class/role/policy/creds) is hub-owned and lands at slice 10; this
slice only mirrors what the agent observes.

### Added
- **`hostReportPayload.StorageTargets`** (`internal/api/handler.go`) — a full mirror of the
  agent's `hub.StorageTarget` wire contract (name/type/durable_id/state/reachable/usage/
  content/mount/class_hint/role/`thin_pool`/`smart`). The targets are persisted verbatim in
  the existing `report_json` row (no schema change); the handler counts them and logs a
  `[WARN]` when any are `disconnected` (the storage analog of host-down visibility).
- **`testdata/host-report.golden.json`** — updated to carry two populated `storage_targets`
  (an lvmthin with `thin_pool`, a usb), kept **byte-identical** with felhom-agent's copy.
- **`TestHostStorageTarget_GoldenContract`** — the hub half of the bidirectional key-set test:
  round-trips the golden's `storage_targets[0]` through the mirror struct and asserts the key
  set matches exactly (no missing/extra fields vs the agent). `TestHostReport_GoldenContract`
  also now asserts the targets are persisted + parse back.

### Notes
- Backward-compatible: an older agent that sends `storage_targets: []` (or omits it) is
  accepted unchanged. The legacy controller report path is untouched (frozen until slice 10).

## Repo docs — no hub version change (2026-06-08)

### Changed
- **Reflowed `felhom.eu/CLAUDE.md`** — removed hard mid-paragraph line wraps (prose, list items, blockquotes now single-line); tables untouched; rendered output unchanged.
- **Unified the REPORT/CHANGELOG convention**: this repo's `REPORT.md` switches from *append/cumulative* to **overwrite-latest** (uniform with the sibling repos); `CHANGELOG.md` (this file) stays the cumulative log, newest on top. Updated `REPORT.md`'s header note accordingly (existing sections retained as history). Added an explicit **no-secrets** rule. No hub code change → no version bump.

## v0.7.1 (2026-06-08)

### Changed
- **`/host-report` rejects oversize bodies explicitly with 413** (`handler.go`) instead of silently truncating at the 4 MiB `LimitReader` cap. Reads one byte past `maxHostReportBytes` and returns `413 Payload too large` — a truncated-but-valid JSON could otherwise be accepted as a partial report (silently dropping guests from the mirror). The controller `handleReport` 1 MiB path is **unchanged** (frozen until slice-10 cutover).

### Added
- **Cross-repo contract fixture** `hub/internal/api/testdata/host-report.golden.json` (byte-identical with felhom-agent's copy) + `TestHostReport_GoldenContract` — POSTs the golden through the real `handleHostReport` and asserts 200 + denorm (`guest_total`/`guest_running`/`cloudflared_status`) + both guests upserted, proving `hostReportPayload` still extracts the contract from the real shape. Duplicated contract (no shared types module yet); revisit at slices 5/6.

## v0.7.0 (2026-06-08)

### Added — host-domain ingest (slice 3, additive; controller path untouched)
- **New tables** `hosts`, `guests`, `host_reports` (`store.go migrate()`, idempotent). Full schema now, including columns **inert until slice 10** (`hosts.desired_json`/`desired_generation`/`dr_record_json`, `guests.api_key`/`desired_spec_json`) so the cutover needs no `ALTER`. Nothing reads/writes the inert columns this slice.
- **`POST /api/v1/host-report`** — the agent's heartbeat. Per-host Bearer auth; 4 MiB body; persists the full report + denormalized fields (cpu/mem/disk %, guest counts, cloudflared status); upserts each guest's **reality** columns (`guest_id = "<host_id>/<vmid>"`, hub-derived); returns the control envelope `{status, poll_interval_seconds:900, blocked, desired_generation:0, has_signed_ops:false}` (`blocked` reflects the customer's status; the latter two are reserved/placeholder for slice 4).
- **Per-host key auth** — `checkAuthHost` (Bearer → host → customer), added alongside the unchanged `checkAuthCustomer`. Global key remains a bootstrap fallback.
- **`POST /api/v1/admin/hosts`** — **PROVISIONAL** global-key-only host mint (host_id + per-host api_key); the slice-3 bootstrap until enrollment (slices 7–8) replaces it.
- **Host dead-man's-switch** — `monitor.HostStalenessChecker` over `host_reports`, emitting `host_stale`/`host_down`/`host_recovered` (30m/60m), attributed to the host's customer; registered in `allowedEventTypes`; wired in `cmd/hub/main.go` on the existing 60s ticker. A deliberate **sibling** of the controller `StalenessChecker` (both run until slice 10).
- **Store methods**: `GetHostByAPIKey`, `GetHost`, `ListHosts`, `UpsertHost`, `SaveHostReport`, `UpsertGuestFromReport` (preserves inert columns on conflict), `GetHostStaleness` (skips never-reported hosts), `GuestID`. `Prune` now also prunes `host_reports` (same retention).
- **Tests** (new, hermetic): store, auth (`checkAuthHost`), ingest (valid+envelope+denorm, host_id mismatch→403, unknown-host-under-global→400, blocked→true, oversize→400), admin mint (non-global→403, unknown customer→400, mint+round-trip), host staleness transitions.

### Unchanged (explicit)
- The controller path — `/api/v1/report`, `reports`, `customer_configs`, `checkAuthCustomer`, the existing staleness/deadline checkers — is untouched and still green. The old controller and the new agent report in parallel during slices 3–9; the schema/auth cutover is **slice 10**.

## v0.6.2 (2026-02-26)

### Added
- **Infra backup GFS retention** — New `infra_backup_versions` table stores multiple backups per customer. GFS pruning keeps: all from last 24h, latest per day (7 days), latest per week (4 weeks), latest per month (3 months) — ~14 versions max per customer
- **`GET /api/v1/infra-backup/{id}/versions`** — Returns metadata list of all retained backup versions (date, stack names, disk count) for a customer. Bearer auth.
- **Recovery version selection** — `GET /api/v1/recovery/{id}?version=ID` fetches a specific backup version instead of latest. Response now includes `backup_versions` array with all available versions.
- **Dashboard backup history** — Customer detail page "Infra Backup" card shows version count and collapsible history table (date, apps, disks)

### Changed
- **`SaveInfraBackup()`** — Now INSERTs a new row instead of upserting, preserving history. Automatically prunes old versions via GFS algorithm.
- **One-time migration** — Existing data from `infra_backups` table is copied to `infra_backup_versions` on first startup

## v0.6.1 (2026-02-25)

### Added
- **Delete issues from app detail page** — Known Issues table now has per-row checkboxes with "Delete Selected" and "Delete All Issues" buttons; keeps telemetry data (memory trends, etc.) intact
- **`DELETE /apps/{appName}/delete-issues`** — New POST endpoint supporting `action=selected` (with `issue_ids` form values) and `action=all`

### Fixed
- **Hub-side fingerprint hardening** — `fingerprintIssue()` now strips ANSI escape codes, ISO/syslog timestamps, and lowercases before truncating to 100 chars. Prevents duplicate issue rows when messages differ only by embedded timestamps.

## v0.6.0 (2026-02-25)

### Added
- **Geo-restriction display** (`customer_unified.html`) — New "Geo-korlátozás" section on customer detail pages showing: enabled/disabled status, allowed countries, per-app overrides, last sync time, and sync errors. Only visible when the controller reports geo_restriction data.
- **"Összes geo-korlátozás eltávolítása" button** — One-click removal of all `[felhom-geo]` Cloudflare WAF rules. The Hub calls the Cloudflare API directly (bypasses potentially blocked tunnel), then retries notifying the controller in background (every 30s for up to 10 min) to disable geo in its settings.
- **Cloudflare unblock client** (`internal/cloudflare/unblock.go`) — Minimal Cloudflare API client for deleting geo-restriction WAF rules. Resolves zone ID, finds the `http_request_firewall_custom` ruleset, and deletes rules with `[felhom-geo]` description prefix.
- **`POST /customers/{id}/geo/disable`** route — CSRF-protected endpoint for the geo-disable action.

### Removed
- **Legacy Monitoring UUIDs** — Removed the "Monitoring UUIDs" section from the config form (`config_form.html`), UUID form-field handling from `buildConfigJSON()`, UUID import from `handlePullConfig()`, volatile key entries for `monitoring.ping_uuids.*`, and the commented-out `ping_uuids` section from `controller.yaml.default`. Monitoring is fully handled by the Hub event system since v0.3.0.

## v0.5.0 (2026-02-25)

### Added
- **Configuration page** (`GET /configuration`) — New "Configuration" tab in the web UI with asset management controls. Displays asset file count, manifest generation timestamp, and a "Refresh Assets from Image" button.
- **Manual asset re-seed** (`POST /configuration`, action=`refresh_assets`) — Re-reads the baked-in seed directory, compares SHA-256 checksums with PVC assets, and updates changed files. Rebuilds the manifest afterward. Controllers pick up changes on their next daily sync.
- **`ReSeed()` method** (`internal/assets/assets.go`) — Public method for triggering asset re-seed + manifest rebuild from the web UI.

### Changed
- **Asset seeding: `seedIfEmpty()` → `seedOrUpdate()`** (`internal/assets/assets.go`) — On startup the Hub now compares SHA-256 checksums between the image seed directory and the PVC, updating any changed files instead of only seeding into an empty directory. This means redeploying the Hub image with updated assets automatically propagates them without PVC deletion.
- **`isAssetFile()` expanded** — Now also matches `*-favicon.svg` and `*-favicon.ico` patterns, allowing branding assets like `felhom-favicon.svg` in the manifest.
- **`RebuildManifest()` refactored** — Internal logic extracted to `rebuildManifestLocked()` for reuse by `ReSeed()`.
- **Web Server struct** — Added `assetsMgr` field and `SetAssetManager()` method. Wired in `main.go`.
- **All templates translated to English** — The "Alkalmazások" nav link and telemetry pages (apps.html, app_detail.html, customer_unified.html telemetry section) are now in English, consistent with the rest of the Hub UI.
- **Navigation updated** — All templates now show four tabs: Dashboard, Customers, Apps, Configuration.

## v0.4.1 (2026-02-23)

### Added
- **Per-app telemetry reset** (`store/telemetry.go`, `web/apps.go`) — New "Telemetria törlése" button on the app detail page that deletes all telemetry records and known issues for the selected app. Useful after major app updates when old data is no longer representative. Includes confirmation dialog and flash notification.
- **`DeleteAppTelemetry()`** and **`DeleteAppIssues()`** store methods (`store/telemetry.go`) — Delete all telemetry/issue rows for a specific app_name.
- **`POST /apps/{name}/reset-telemetry`** route (`web/server.go`) — CSRF-protected endpoint that triggers the reset and redirects back with flash message.

## v0.4.0 (2026-02-23)

**App Telemetry & Analytics Dashboard**

### Added
- **`app_telemetry` and `app_log_issues` SQLite tables** (`store/store.go`) — store per-app resource metrics and deduplicated log issues reported by v0.28.0+ controllers.
- **`internal/store/telemetry.go`** — New store methods: `SaveAppTelemetry`, `GetFleetAppSummary` (with P95 memory calculation), `GetAppTelemetryHistory`, `GetAppCustomerBreakdown`, `GetCustomerAppSummary`, `GetAppIssues`, `GetRecentIssuesAllApps`, `PruneAppTelemetry`, `PruneStaleIssues`. New types: `AppTelemetryRecord`, `FleetAppSummary`, `AppTelemetryPoint`, `AppCustomerStats`, `CustomerAppSummary`, `AppIssue`.
- **`/api/v1/report` handler update** (`api/handler.go`) — After saving the standard report, parses the optional `app_telemetry` JSON field and persists it. Backward-compatible: old controllers (no `app_telemetry` key) are unaffected.
- **Fleet app list page** (`GET /apps`) — Hungarian-language dashboard showing all deployed apps fleet-wide with deployment count, avg/P95 memory, catalog estimate/limit accuracy, error/warning badges. Sortable columns, 24h/7d/30d period selector.
- **Per-app detail page** (`GET /apps/{name}`) — Memory trend Chart.js chart (avg + peak, with catalog limit line), per-customer breakdown table, known log issues table (severity, message, occurrence count, affected customers). Includes suggested mem_limit from P95×1.2 rounded to 32M.
- **Customer detail page telemetry section** (`customer_unified.html`) — New "Alkalmazás telemetria" card with per-app memory (current/avg/peak) and log error/warning counts linking to /apps/{name}.
- **Chart.js** (`static/chart.min.js`) — Embedded from controller build, served at `/static/chart.min.js`.
- **"Alkalmazások" nav link** — Added to header navigation across all templates.
- **New CSS** (`style.css`) — `.badge`, `.badge-error`, `.badge-warn`, `.summary-cards`, `.summary-card`, `.chart-container`, `.period-selector`, `.period-btn`, `.accuracy-dot`, `.mem-ok/warn/danger`, `.data-table` styles.
- **Telemetry pruning** (`cmd/hub/main.go`) — `pruneAll()` now also prunes app_telemetry rows older than 90 days and stale log issues not seen in 30 days.

### Changed
- **`internal/web/apps.go`** (new file) — `handleApps`, `handleAppDetail`, `parsePeriod`, `sortFleetSummary`, `aggregateHistoryForChart`, `parseLimitMB`, `memoryColor`, `accuracyClass`, `getCSRFToken` helper functions.
- **`internal/web/server.go`** — Added routes for `/apps`, `/apps/{name}`, `/static/chart.min.js`. Added `memoryColor`, `accuracyClass`, `gt` template functions.
- **`internal/web/embed.go`** — Added `//go:embed static/chart.min.js` directive.

## v0.3.7 (2026-02-21)

**Asset management API**

- New `internal/assets` package: manages app assets (logos, screenshots) on Hub PVC (`/data/assets/`) with automatic seeding from baked-in image copy on first run.
- Two new authenticated API endpoints for controllers to sync assets:
  - `GET /api/v1/assets/manifest` — returns JSON manifest with filenames + SHA-256 checksums
  - `GET /api/v1/assets/file/{filename}` — serves individual asset files
- Dockerfile updated to `COPY assets/ /usr/share/felhom/assets-seed/` for first-run seeding.
- Build script syncs website assets (`*-logo.{svg,png}`, `*-screenshot-*.webp`) into Docker build context.

## v0.3.6 (2026-02-21)

**Human-friendly retrieval passwords**

- Retrieval passwords now use Hungarian word passphrases (e.g. `áldás-plazmid-palánta-süvítve-pócgém`) instead of 64-char hex strings.
- Embedded 29K+ curated Hungarian word list (`hungarian.txt`) via go:embed; 5-word passphrases give ~74 bits of entropy.
- New `configgen.RandomPassphrase(wordCount)` function; all 3 retrieval password generation sites updated.
- API keys remain as hex (machine-to-machine, never typed by humans).

## v0.3.5 (2026-02-21)

**Recovery Endpoint & Customer Standing**

- New `GET /api/v1/recovery/{customer_id}` endpoint: returns both generated controller.yaml and infra backup in a single response for disaster recovery. Auth via `X-Retrieval-Password` header (same as config retrieval).
- Report response now includes `customer_blocked: true` when customer status is "blocked" — allows controllers to detect standing and enter limited mode.

## v0.3.4 (2026-02-20)

- Rename version labels: "Current version" → "Controller version", "Latest version" → "Registry latest".

## v0.3.3 (2026-02-20)

**Bugfixes**

- Fix double "v" prefix in controller version display (showed "vv0.21.1" instead of "v0.21.1").
- Skip deprecated `monitoring.ping_uuids.*` keys in config diff comparison (added to volatile keys).

## v0.3.2 (2026-02-20)

**Hub Version Display**

- Show Hub version in footer of all pages via `hubVersion` template function.
- `web.New()` now accepts `version` parameter (4th arg) — set via ldflags at build time.

## v0.3.1 (2026-02-20)

**Config Diff Display + Pull Config**

- **Value-based config comparison**: Replaced broken SHA256 hash comparison with semantic YAML comparison. Both configs are parsed into maps, flattened to dot-notation keys, and compared by value. Ignores key ordering, whitespace, comments, and volatile fields (`web.session_secret`). Shows actual diff count on customer page ("⚠ Config mismatch — N differences").
- **Config diff endpoint** (`GET /customers/{id}/config-diff`): Fetches live YAML from controller via new `GET /api/config` endpoint, generates Hub YAML via `configgen.Generate()`, returns JSON with per-key diffs (key, hub value, controller value, status). Sensitive values (tokens, passwords, secrets) are masked.
- **Pull Config** (`POST /customers/{id}/pull-config`): Reverse of Push Config — imports controller's current config into the Hub. Extracts identity fields (name, domain, email) and override fields (infrastructure tokens, git credentials, monitoring UUIDs). Preserves existing APIKey and RetrievalPassword.
- **Diff display UI**: "Show Diff" button on customer page expands a table showing all key-value differences with color-coded rows (yellow=changed, blue=hub-only, orange=controller-only).
- **Pull Config button**: Added next to existing "Push Config" with confirmation dialog.

## v0.3.0 (2026-02-20)

**Hub Monitoring Takeover — Event System, Dead Man's Switch, Notifications**

Replaces external Healthchecks.io with a Hub-native event system. The Hub becomes the single source of truth for all customer monitoring, event tracking, dead man's switch alerting, and notification delivery.

### Phase 1 — Event System
- **`events` table** in SQLite: stores all events with customer_id, event_type, severity, message, details_json, source, timestamp
- **Indexes**: `idx_events_customer_created` (customer + time DESC), `idx_events_type` (type + time DESC)
- **Store methods**: `SaveEvent`, `GetRecentEvents`, `GetEventsByType`, `GetLatestEventByType`, `GetAllRecentEvents`, `CountEventsBySeverity`, `PruneEvents`, `GetActiveCustomerIDs`
- **`POST /api/v1/event`** endpoint: accepts structured events from controllers, validates event_type against 27 allowed types, validates severity (info/warning/error), stores in DB
- **Enhanced auth**: `checkAuthCustomer()` validates per-customer API keys match the customer_id in payload; global key bypasses ownership check
- **Prune**: events pruned alongside reports at 04:30 Budapest time

### Phase 2 — Dead Man's Switch
- **Staleness checker** (`internal/monitor/staleness.go`): runs every 60s, detects when controllers stop reporting
  - ok→stale (>30min): inserts `node_stale` warning event
  - any→down (>60min): inserts `node_down` error event
  - stale/down→ok: inserts `node_recovered` info event
  - Skips blocked customers, no false alerts on startup
- **Backup deadline checker** (`internal/monitor/deadline.go`): runs daily at 05:00 Budapest
  - Detects missing `backup_completed` events since midnight → inserts `expected_backup_missed` error
  - Detects missing `db_dump_completed` events → inserts `expected_dbdump_missed` error
  - Grace: skips customers with `node_down` state
- **`scheduleDaily()`** helper: goroutine that sleeps until target time (Europe/Budapest), runs function, loops
- **`/healthz`** enhanced: returns 503 if SQLite Ping fails

### Phase 3 — Notification System
- **Dispatcher** (`internal/notify/dispatcher.go`): processes events and sends emails via Resend API
  - **Operator channel**: English emails to operator for warning/error events, 1h cooldown per customer:eventType
  - **Customer channel**: Hungarian emails per event_type, respects customer preferences (enabled_events, cooldown_hours), blocked customers skipped
  - **Test bypass**: `test` event type skips cooldown/preferences, sends directly to customer email
- **Email templates** (`internal/notify/templates.go`): operator (concise English), customer (Hungarian per event type with complete message table)
- **Cooldown tracking**: in-memory maps with per-customer:eventType granularity
- **`customer_notifications` table**: added `cooldown_hours` column (default 6)
- **`notification_log` table**: added `channel` column (operator/customer)
- Wired into `/api/v1/event` handler and staleness/deadline checkers

### Phase 4 — Hub UI
- **Events section** on customer detail page: last 50 events, severity filter buttons (All/Errors/Warnings/Info), colored severity badges
- **Dashboard badges**: error+warning count in last 24h per customer, clickable to customer events
- **Notification log**: shows channel column (operator/customer) in customer detail page
- **Config form**: Monitoring UUIDs section marked as "Legacy" with deprecation notice, collapsed by default

### Phase 6 — Config Cleanup
- **`controller.yaml.default`**: `monitoring.ping_uuids` section commented out (deprecated)
- **`buildConfigJSON`**: only writes `ping_uuids` to config JSON if user explicitly provides UUID values (new configs get none)

---

## v0.2.2 (2026-02-20)

**Config Hash Comparison**

- **Config sync status** on unified customer page: compares SHA256 hash of controller's
  `controller.yaml` (from report payload) against Hub-generated YAML. Shows "In sync",
  "Config mismatch", or "Unknown" (controller needs v0.20.0+ to report hash).
- Visible in the Controller Update section next to Push Config button.

---

## v0.2.1 (2026-02-20)

**Unified Customer Management**

All customer views consolidated into a single page. New management features: blocked status,
dashboard merge, config push, and auto-config creation.

### New features

- **Unified customer page — `/customers/{id}`:**
  - Single page showing both configuration info and live report data
  - Replaces separate `/configs/{id}` (config detail) and `/customers/{id}` (report detail) pages
  - Shows config management (credentials, setup commands, YAML preview) when config exists
  - Shows "Create Config" button for manual (report-only) customers
  - Old `/configs/{id}` URLs redirect to `/customers/{id}`

- **Dashboard shows pending customers:**
  - Customers with config but no reports appear on dashboard with "PENDING" status
  - All metric columns show "—" for pending customers

- **Blocked/Banned status:**
  - Customers can be blocked via button on detail page
  - Blocked customers hidden from Dashboard
  - Reports still accepted (prevents controller retry loops) but notifications suppressed
  - "BLOCKED" badge shown on Customers list and detail page
  - One-click unblock button

- **Config push to controller:**
  - "Push Config" button on unified page (visible when controller URL known)
  - Generates YAML and POSTs to `{controller_url}/api/config/apply`
  - Note: requires controller v0.20.0+ with config apply endpoint

- **Auto-create config from report data:**
  - "Create Config" button on manual customer pages
  - Pre-fills customer name from report, generates credentials
  - Redirects to edit form for additional fields

### Changes

- Customers list: all rows now link to `/customers/{id}` (unified page)
- Config badges: new MANAGED/MANUAL/BLOCKED pill-style badges
- `customer_configs` table: added `status` column (active/blocked)
- Status functions handle "pending" and "blocked" status values

---

## v0.2.0 (2026-02-20)

**Customer Configuration Management**

New "Configurations" section for pre-provisioning customer nodes. Operators can configure
customer settings in the Hub web UI, then `docker-setup.sh` downloads a ready-made
`controller.yaml` — reducing deployment to a customer ID and password.

### New features

- **Web UI — `/configs` pages:**
  - List all customer configurations in a table
  - Create new configuration: customer identity, infrastructure secrets (CF tunnel/API tokens),
    git sync credentials, monitoring UUIDs — organized in collapsible sections
  - Detail page: shows credentials (retrieval password, per-customer API key) with copy-to-clipboard,
    setup commands (`docker-setup.sh` and `curl`), live YAML preview
  - Edit and delete configurations
  - Navigation tabs (Dashboard / Configurations) on all pages

- **Config retrieval API — `GET /api/v1/config/{customer_id}`:**
  - Authenticated via `X-Retrieval-Password` header (separate from Bearer token)
  - Generates complete `controller.yaml` by deep-merging template with customer overrides
  - Template sourced from `controller.yaml.example` (fetched from Gitea repo periodically)
  - Falls back to embedded default template if fetcher not configured

- **Per-customer API keys:**
  - Each customer config gets its own API key (auto-generated, 64 hex chars)
  - Controllers can authenticate with per-customer key instead of the shared global key
  - Backward compatible — global `report_api_key` continues to work alongside per-customer keys

- **YAML generation (`internal/configgen` package):**
  - Deep-merge of template + customer-specific overrides
  - Programmatic injection: customer identity, hub config, session secret
  - Shared by both API handler and web UI preview

- **Template fetcher (background goroutine):**
  - Periodically fetches `controller.yaml.example` from Gitea (configurable interval)
  - Requires `registry.username` + `registry.token` in hub.yaml
  - Falls back to `go:embed` default template when not configured

- **Data layer:**
  - New `customer_configs` SQLite table
  - 6 CRUD methods: Save, Get, List, Delete, GetByAPIKey, UpdateRetrievalPassword

### Configuration

New `registry` section in `hub.yaml`:

```yaml
registry:
  image: "gitea.dooplex.hu/admin/felhom-controller"
  username: ""               # Gitea credentials (for version checker + template fetcher)
  token: ""
  check_interval: "6h"
  template_interval: "1h"   # How often to refresh controller.yaml.example
```

### Files added

- `internal/configgen/configgen.go` — shared YAML generation package
- `internal/web/configs.go` — web handlers for config CRUD
- `internal/web/templatefetcher.go` — background template refresh
- `internal/web/controller.yaml.default` — embedded fallback template
- `internal/web/templates/configs.html` — config list page
- `internal/web/templates/config_form.html` — create/edit form
- `internal/web/templates/config_detail.html` — detail + credentials page

### Files modified

- `internal/store/store.go` — customer_configs table + CRUD methods
- `internal/api/handler.go` — config retrieval endpoint, per-customer auth, `ConfigTemplateProvider` interface
- `internal/web/server.go` — `/configs/*` routes, `SetTemplateFetcher()`
- `internal/web/embed.go` — embedded default template
- `internal/web/templates/dashboard.html` — navigation bar
- `internal/web/templates/customer.html` — navigation bar
- `internal/web/templates/style.css` — form, nav, button, credential styles
- `cmd/hub/main.go` — template fetcher wiring, `TemplateInterval` config
- `configs/hub.yaml.example` — registry section

---

## v0.1.8 (2026-02-16)

- Controller update trigger: "Update" button on customer detail page calls controller's self-update endpoint
- Registry version checker: background goroutine checks Gitea registry for latest controller image tag
- Update available indicator on customer detail page

## v0.1.7 (2026-02-15)

- Infrastructure backup endpoints for disaster recovery (POST + GET `/api/v1/infra-backup`)

## v0.1.6 (2026-02-14)

- Handle disabled reporting status
- Storage labels display
- Date in history table

## v0.1.5 (2026-02-13)

- Notification preferences sync endpoint (`POST /api/v1/preferences`)
- Notification display on customer detail page

## v0.1.4 (2026-02-12)

- Resend API key support for email notifications
- Notification endpoint (`POST /api/v1/notify`)

## v0.1.3 (2026-02-11)

- Customer detail page: system info, storage bars, container table
- 24h history graphs

## v0.1.2 (2026-02-10)

- Dashboard auto-refresh (60s cycle)
- Status logic (green/yellow/red based on report age + health)

## v0.1.1 (2026-02-09)

- Basic dashboard with customer overview table
- Report ingest API

## v0.1.0 (2026-02-08)

- Initial release: SQLite store, report API, basic web dashboard