hub v0.84.0 — break-glass console credential on the host page

The credential existed and was not reachable when it was wanted. Every box has
had a strong random root@pam password since TASK G1, vaulted in the hub at day 0
and used for real during the sshd incident — but the only way to read it back was
a hand-written curl carrying the global operator key, a secret kept out-of-band.
In practice the PVE web console on a demo box felt locked.

The host page grows a Console access card: presence + username + set_at by
default, Reveal fetches the plaintext on demand for 60 s with a Copy button.
Masking clears the JS variable, and also fires on a second click and on
visibilitychange. A host with nothing vaulted says so, and says why.

The secret is NEVER rendered into the page, and that constraint shapes the
change. The render path uses a new store.GetHostRecoveryMeta whose struct and
SELECT both omit the secret column, so it is structurally incapable of carrying
one. The plaintext crosses the wire only in the response to POST
/hosts/{id}/reveal-recovery-credential (Cache-Control: no-store, CSRF-gated at
the ServeHTTP level; POST precisely so that gate applies and so no secret is
retrievable by URL alone). Deliberately NOT the customer page's data-secret
widget, which embeds the plaintext on every load.

A delivered reveal writes one recovery_credential_revealed event on the host's
customer timeline (info, source hub, Hungarian) via SaveEvent alone — no
dispatcher, nobody emailed, the log_tail_requested shape. Two reveals write two
events: the register records accesses, not states. A 404 is not an access. An
unbound host reveals fine and writes no event; the [INFO] hub line, carrying the
username and a length only, is then the record.

The global-key API path is untouched by design — it is the route for when the
hub UI itself is broken, and coupling it to the session layer would delete the
independence that makes it a fallback.

Recorded as a real trade: the hub session password alone now unlocks console root
fleet-wide, where retrieval previously also needed the global key. Accepted for a
single-operator, HU-geo-fenced hub that already stores these passwords in
plaintext at rest (CONTEXT.md ruling S-4). The plaintext-at-rest half is filed as
R-133 — every hub DB backup is a fleet-wide console-credential dump.

Tests 550 -> 559; four red-proofs (page leak, audit event, CSRF gate, route
order) each run, observed failing, and reverted. The route-order proof is a seam
test driving ServeHTTP: a handler-level test cannot see that defect, because the
handler is correct and simply never runs.

documentation/architecture/00-capability-map.md

@@ -117,7 +117,7 @@
 | Forgot dashboard password → instant reset code | controller v0.123, hub | **PROVEN-LIVE** | `DRILL-day0-take2-2026-07-12` F-15 (live re-run of the exact failure path: hash applied 1s after request, code accepted first try) | |
 | Multiple household users / per-person accounts | — | **MISSING** | — | Single dashboard password; acceptable for alpha → R-15 |
 | WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | `SPIKE-oob-wg-operator-peer-2026-07-05`, `SPIKE-felhom-sshd-2026-07-05` | Mutual-repair desired-state arc not built → R-13 |
-| Break-glass management-plane recovery | agent v0.71, hub v0.34 | **IMPLEMENTED** | `runbooks/break-glass.md` | |
+| Break-glass management-plane recovery | agent v0.71, hub v0.84 | **IMPLEMENTED** | `runbooks/break-glass.md` | hub v0.84.0 adds an **operator-SESSION** retrieval path (host page → Console access → Reveal; `POST /hosts/{id}/reveal-recovery-credential`, CSRF-gated, writes a customer-visible `recovery_credential_revealed` event) beside the pre-existing **global-key** one (`GET /api/v1/admin/hosts/{id}/recovery-credential`), which is untouched and stays the route for when the hub UI itself is down. Still IMPLEMENTED, not PROVEN-LIVE: the UI path has not been exercised on a real lockout, and that the revealed password authenticates at `:8006` is operator-verified only. The vaulted secret is plaintext at rest → **R-133** |
 
 ## F. Notifications & monitoring
 

documentation/backlog/OPEN-ITEMS.md

@@ -70,6 +70,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
 | **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
 | **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
 | **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
+| **R-133** | **The vaulted break-glass console credential is PLAINTEXT AT REST — every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim, so any copy of the SQLite DB (Longhorn snapshot, PBS backup of the hub PVC, a hand-taken copy during a diagnosis) carries root console access to every Felhom host in one file | **READY (M) — NEW 2026-07-31** | — | **The deferred leg of hub v0.84.0** (Console access card), filed separately because v0.84.0 changed only WHO can retrieve the secret, never how it is stored. v0.84.0 makes it more worth doing, not more broken: retrieval now rides the hub SESSION, so the DB and the login password are jointly the whole protection (ruling **S-4**, `CONTEXT.md`). Fix shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held outside the DB** — the hub already proves it can hold something it cannot itself read (escrow blobs), and that contrast is the argument. Two constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the whole point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Would flip the capability-map row **"Break-glass management-plane recovery"**, which today reads IMPLEMENTED with this as its caveat | CC |
 
 ## Why the TOP READY rows rank this way
 

documentation/backlog/ROADMAP.md

@@ -110,6 +110,7 @@
 | R-107 | **No offsite action unpacks the named-volume tars Tier-3 captures on every run.** `ReconstituteFromOffsite` skips the unit outright (`internal/backup/offbox_reconstitute.go:284-289`); `PlaceOffsiteRestore` places it only when the live unit is ABSENT (`internal/backup/offbox_restore.go:352-356`). The DB **is** replayed, from the scratch unit's `db-dumps/` | M | READY — 2026-07-28 | 52 of 53 catalog templates keep data in Docker named volumes, so for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists (full restore → place → Tier-1 restore, which does unpack tars) but **no single action does it and no UI routes it** — the R-103 shape one tier over. Sibling of R-102: both are "captured and never read". Flips: the offsite-restore row in map §C; `07` §6.3, §7.2 |
 | R-108 | ~~Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5~~ | M | **SHIPPED** — controller v0.187.0, 2026-07-30 | **D5 UNBLOCKED.** One fail-closed predicate (`Settings.RefuseAsAppNamespace`) consulted by every placement surface, so no `backups/` tree can exist inside the share-root bind. The bind was deliberately NOT narrowed — it is load-bearing (`:rslave` automount wake, 2026-07-22 probe) and unscopable (apps on a share store at `<share>/<app>`; creating a `userdata/` layer would write Felhom convention onto a customer's NAS). **Five surfaces, not the four this row named** — decommission-with-migrate guarded only its SOURCE; and the boundary is the deploy POST, not the dropdown. Zero apps on network storage fleet-wide, so nothing stranded. The `.fab`-onto-NAS "second effect" is split out as **R-126**. Flipped: `07` §7.3 (D5 → unblocked), §10.1 (CLOSED), `00-capability-map.md` §D (new PROVEN-LIVE row). `audits/R108-network-app-namespace-2026-07-30.md` |
 | R-127 | **`data_key: true` is unreliable (4+ encryption keys unflagged, contradicting the catalog's own labels), and O4 can regenerate a DB password that no longer matches the restored data directory** | S/M | READY — NEW 2026-07-30 | Found by **D5's Part 0**, and the reason D5's boundary became `type: secret` rather than `data_key`. Leg (a): flag the missing keys (catalog-only) + pin flag-vs-label agreement; the residual risk after D5 is that the **fail-closed gate** keys on `data_key`, so an unflagged key missing from both sources lets the restore proceed onto undecryptable data. Leg (b): a regenerated DB password is silently wrong — `POSTGRES_PASSWORD` is ignored once PGDATA is non-empty, so the app cannot authenticate while the replay still succeeds over the local trust socket (proven live on `postgres:16-alpine`). v0.188.0 corrected the false "stored data is unaffected" WARN but added no guard. Flips: `07` §7.4 |
+| R-133 | **The vaulted break-glass console credential is PLAINTEXT AT REST, so every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim (`hub/internal/store/host_recovery.go` — *"a hub-held secret, operator-retrievable (NOT zero-knowledge like escrow)"*), so anything that copies the SQLite DB — a Longhorn snapshot, a PBS backup of the hub PVC, a hand-taken copy during a diagnosis — carries root console access to every Felhom host in one file, with no second factor and no key to withhold | M | READY — NEW 2026-07-31 | **The DEFERRED LEG of hub v0.84.0**, filed as its own ID because v0.84.0 changed only WHO can retrieve the secret, never how it is stored — the at-rest shape predates it and is untouched by it. **v0.84.0 made it more worth doing, not more broken:** putting retrieval behind the hub session means the hub login password alone now unlocks console root fleet-wide, so the DB and the login are the whole of the protection. Shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held OUTSIDE the DB** (k8s Secret / out-of-band file, the way `manifests/` already keeps the bearer out of git), so a DB copy is opaque the way escrow blobs already are — the contrast is the argument, since the hub already proves it can hold a secret it cannot itself read. Constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the entire point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Flips: `architecture/00-capability-map.md` **"Break-glass management-plane recovery"** — the row that today reads IMPLEMENTED with a plaintext-at-rest caveat |
 | D5 | ~~**Move app secrets into the LOCAL recovery unit** so Tier-1/Tier-2 restore stop needing the guest and stop needing R~~ | M | **SHIPPED + PROVEN-LIVE** — controller v0.188.0, 2026-07-30 | **The arc's architectural centrepiece. Tier-1/2 no longer depend on the whole-guest tier — a customer needs the DRIVE AND NOTHING ELSE.** Part 0 tested this row's own premise and **rejected** it: data-keys-only is both insufficient and unsafe, because `data_key` is unreliable (→ **R-127**) and a DB password is not resettable in practice (`POSTGRES_PASSWORD` is ignored once PGDATA is non-empty, so a regenerated value leaves the app unable to reach its own restored rows while the dump replay still reports success — proven on `postgres:16-alpine`). **Operator ruling: `type: secret` travels (45 fields), `type: password` never (7) plus a code register (`vaultwarden/ADMIN_TOKEN`); plaintext, because withholding the internet-reachable class is what licenses it — the two are coupled.** `stacks.PortableSecretEnvVars` is the single boundary; the register is code, not a catalog flag (R-97a). **Precedence: the UNIT WINS** (its secrets match the data being restored, not merely the newest), pinned both directions. Fail-closed data-key gate UNCHANGED. Manifest schema 2; schema-1 units still restore. Proven live on a scratch drill guest: AdventureLog restored with the guest `app.yaml` moved aside (`secrets recovered=2/2`, 27.6 s) and **the app read the seeded row over TCP with its own credential**; Grafana's admin password withheld with **0 hits** across the backup namespace. 4 red-proofs each verified to land. `audits/D5-drive-alone-restore-2026-07-30.md` Flips `07` §3/§7.1/§7.3/§7.4/§8/§10.1 + a new capability-map row |
 | R-126 | **A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS.** `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | S | READY — 2026-07-30 | Split out of R-108 on its closure. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (`07` §7.3 records the reasoning). Fix = filter network paths from the export destination list, or force the bundle password when the destination is a share. Flips: `07` §5 |
 | R-109 | ~~The DR recipe records no backup target~~ | XS | **SHIPPED** — agent v0.118.1 + hub v0.83.0, 2026-07-30 | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` (the scheduler's own function) + the mountpoint that separates `/mnt/hdd_1` from `/var/lib/vz`; unresolvable is recorded as unresolvable, never defaulted. Required an unscoped HUB half: `AssembleDRRecipe` allow-lists top-level keys → **R-122**, a fourth recipe-completeness defect that had already shipped (`offsite_restic` dropped for the feature's whole life). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 |

documentation/runbooks/break-glass.md

@@ -39,13 +39,26 @@ Layer 2 heals within ~60s with nobody logged in. If you just saw the KEXINIT-res
 retry SSH. If it's back, check the hub for a `mgmt_plane_healed` warning and go to §4 (find the cause).
 
 ### 3.1 If SSH is still dead — reach the PVE web console (break-glass)
-1. **Retrieve the vaulted root@pam password** (operator/global key):
+1. **Retrieve the vaulted root@pam password.**
+
+   **Primary route — the hub UI (hub v0.84.0):** hub → **Hosts** → the host → the **Console access**
+   card → **Reveal**. The password shows for 60 s with a Copy button. The card also states which user
+   it is for and when it was last vaulted, so a stale copy is visible before you try it. A host with
+   nothing vaulted says so and says why, instead of showing a blank. Needs only the hub login
+   password — no out-of-band key.
+
+   **Fallback — the global-key curl. Use this when the hub UI itself is unavailable** (that is the
+   whole reason it still exists, and why it deliberately has no dependency on the hub's session
+   layer):
    ```
    curl -s -H "Authorization: Bearer <GLOBAL-KEY out-of-band>" \
      https://hub.felhom.eu/api/v1/admin/hosts/<host_id>/recovery-credential
    # → {"host_id":"…","username":"root@pam","password":"…","set_at":"…"}
    ```
    (A per-host key is refused here by design — only the operator key reads a console password back.)
+
+   Either route is an **operator action against a customer's box**, and the UI route records it: see
+   §5.
 2. **Open the PVE web UI** `https://<host-ip>:8006`, log in `root@pam` + that password, realm
    *Linux PAM standard authentication*.
 3. **Node → Shell** (xterm.js root console). Or navigate directly:
@@ -87,6 +100,17 @@ Fix the cause; the three layers are the safety net, not the cure.
   wipes `state.json`, so step 4b re-randomizes — re-run the set-and-vault afterward. Background: on each
   full reinstall this is exactly why `root@pam` "kept changing" until it was pinned.
 - **The password is never logged or filed** — it exists only in the hub vault (retrieve as in §3.1).
+- **Revealing it through the hub UI writes a customer-visible event line** (hub v0.84.0):
+  `recovery_credential_revealed`, info severity, on that host's customer timeline, in Hungarian —
+  *„Az üzemeltető lekérte a géped konzolos hozzáférési jelszavát (távoli hibaelhárítás)."* Nobody is
+  emailed (no dispatcher call), and the register records **accesses**, so two reveals write two
+  lines. Deliberate, for a product sold on data sovereignty; it mirrors how a remote log pull is
+  already recorded. An **unbound** host has no customer to tell — there the hub `[INFO]` log line is
+  the only record. The global-key curl path writes no event.
+- **The hub login password now unlocks console root on every managed box.** Before v0.84.0,
+  retrieval also required the global API key. Accepted for a single-operator, HU-geo-fenced hub that
+  already holds these passwords in plaintext at rest — a convenience trade, and a real one. The
+  plaintext-at-rest half (every hub DB backup is a fleet-wide console-credential dump) is **R-133**.
 - **Least-privilege console user** instead of root@pam was probed as a non-blocking observation: PVE
   has historically gated the node Shell to `root@pam`, so G1 ships the proven root@pam path. Revisit if
   PVE later allows a scoped realm user to open a node Shell.