From 1956e5d390529aaeffd368a79ec2875c904d6218 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 31 Jul 2026 08:19:36 +0200 Subject: [PATCH] =?UTF-8?q?hub=20v0.84.0=20=E2=80=94=20break-glass=20conso?= =?UTF-8?q?le=20credential=20on=20the=20host=20page?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The credential existed and was not reachable when it was wanted. Every box has had a strong random root@pam password since TASK G1, vaulted in the hub at day 0 and used for real during the sshd incident — but the only way to read it back was a hand-written curl carrying the global operator key, a secret kept out-of-band. In practice the PVE web console on a demo box felt locked. The host page grows a Console access card: presence + username + set_at by default, Reveal fetches the plaintext on demand for 60 s with a Copy button. Masking clears the JS variable, and also fires on a second click and on visibilitychange. A host with nothing vaulted says so, and says why. The secret is NEVER rendered into the page, and that constraint shapes the change. The render path uses a new store.GetHostRecoveryMeta whose struct and SELECT both omit the secret column, so it is structurally incapable of carrying one. The plaintext crosses the wire only in the response to POST /hosts/{id}/reveal-recovery-credential (Cache-Control: no-store, CSRF-gated at the ServeHTTP level; POST precisely so that gate applies and so no secret is retrievable by URL alone). Deliberately NOT the customer page's data-secret widget, which embeds the plaintext on every load. A delivered reveal writes one recovery_credential_revealed event on the host's customer timeline (info, source hub, Hungarian) via SaveEvent alone — no dispatcher, nobody emailed, the log_tail_requested shape. Two reveals write two events: the register records accesses, not states. A 404 is not an access. An unbound host reveals fine and writes no event; the [INFO] hub line, carrying the username and a length only, is then the record. The global-key API path is untouched by design — it is the route for when the hub UI itself is broken, and coupling it to the session layer would delete the independence that makes it a fallback. Recorded as a real trade: the hub session password alone now unlocks console root fleet-wide, where retrieval previously also needed the global key. Accepted for a single-operator, HU-geo-fenced hub that already stores these passwords in plaintext at rest (CONTEXT.md ruling S-4). The plaintext-at-rest half is filed as R-133 — every hub DB backup is a fleet-wide console-credential dump. Tests 550 -> 559; four red-proofs (page leak, audit event, CSRF gate, route order) each run, observed failing, and reverted. The route-order proof is a seam test driving ServeHTTP: a handler-level test cannot see that defect, because the handler is correct and simply never runs. --- CONTEXT.md | 15 + REPORT.md | 256 +++++-------- REUSE.md | 1 + .../architecture/00-capability-map.md | 2 +- documentation/backlog/OPEN-ITEMS.md | 1 + documentation/backlog/ROADMAP.md | 1 + documentation/runbooks/break-glass.md | 26 +- hub/CHANGELOG.md | 55 +++ hub/internal/store/host_recovery.go | 28 ++ hub/internal/store/host_recovery_test.go | 37 ++ hub/internal/web/hosts.go | 77 ++++ .../web/hosts_recovery_reveal_test.go | 360 ++++++++++++++++++ hub/internal/web/server.go | 9 + .../web/templates/host_detail_body.html | 90 +++++ 14 files changed, 801 insertions(+), 157 deletions(-) create mode 100644 hub/internal/web/hosts_recovery_reveal_test.go diff --git a/CONTEXT.md b/CONTEXT.md index 4aa4035..96375c4 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -59,6 +59,21 @@ cited as authoritative). They are **decisions, not observations**; the rewrite l compromised hub yields blobs nobody can open, *provided the operator's key is never stored in the hub*. That proviso is why escrow custody is an open decision (`07` §11-A). +**S-4 — the hub session password alone now unlocks console root on every managed box (2026-07-31, hub v0.84.0).** +Retrieving a host's vaulted break-glass `root@pam` credential previously required the **global operator +API key**, a secret distinct from the hub login and kept out-of-band. The `Console access` card on the +host page puts that retrieval behind the **hub session** (`POST /hosts/{id}/reveal-recovery-credential`, +CSRF-gated, audited), so the hub login password is now sufficient. **Accepted deliberately**, and the +reasoning is the ruling: the hub is single-operator and geo-fenced to Hungary, and it already stores +these passwords **in plaintext at rest** — so the DB and the login were already jointly sufficient, and +this is a convenience trade rather than a new exposure. It is a real trade, not a free one, and it is +recorded here rather than only in a changelog. Two things follow. **(a)** The global-key path +(`api/handler.go handleAdminGetRecoveryCredential`) stays untouched and must acquire **no** dependency +on the session layer — it is the route for when the hub UI itself is what is broken, and coupling the +two would delete the independence that makes it a fallback. **(b)** The plaintext-at-rest half is now +the named open item **R-133**: every hub DB backup is a fleet-wide console-credential dump, and the fix +shape is envelope encryption under a KEK held outside the DB, the way escrow blobs are already opaque. + Five decisions were deliberately **left open for the operator** and are recorded, unanswered, in `07` §11: escrow custody · lost-R policy · per-scenario RTO/RPO targets (**none have ever been stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with diff --git a/REPORT.md b/REPORT.md index 5772ece..caf9aa3 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,183 +1,129 @@ -# REPORT — E-2 increment 1: the backup target is now defended and audible (2026-07-29) +# REPORT — Hub v0.84.0: break-glass console credential on the host page (2026-07-31) -**Overwritten** per the standing rule. **E-2 is PARTIALLY shipped** — this increment is Parts 1 + 5 -plus findings E-2b/E-2c. Parts 2, 3, 4 and E-2a remain open (see §6). +## 1. Confirmed baselines actually used -**Live fleet after this run:** hub **0.81.0**, agent **0.112.0** (both boxes), controller **0.184.1** -(both boxes). `peti-felhom` deliberately untouched. +| Repo | `main` @ start | Version before | Version after | +|---|---|---|---| +| felhom.eu | `0a9bd3829d9a4465b032ea78813f55e935fbf88e` ("D5 SHIPPED…", 2026-07-30) | hub **v0.83.0** (`manifests/hub.yaml:128` → `felhom-hub:0.83.0`) | hub **v0.84.0** | ---- +Clean-tree gate at start: `git status --porcelain` empty, `HEAD == origin/main`. **Note:** a second +Claude Code session was working in this shared clone concurrently — see §10. -## 1. Why this order +## 2. Files created / modified -The operator reordered the plan, correctly: **E-2b and E-2c protect a configuration that exists -right now; E-2a builds for boxes that do not.** Both demo boxes are configured, Peti is parked, and -no new installs are happening this week — so Part 3 has no user yet, while E-1's new configuration -had a drive that could be ejected without warning and no event when it went absent. +Created: +- `hub/internal/web/hosts_recovery_reveal_test.go` -## 2. E-2c — the backup drive could be ejected out from under the backup +Modified: +- `hub/internal/store/host_recovery.go` — `HostRecoveryMeta` + `GetHostRecoveryMeta` +- `hub/internal/store/host_recovery_test.go` — Group H +- `hub/internal/web/hosts.go` — `handleHostRevealRecoveryCredential`; `hostDetailData` +3 keys +- `hub/internal/web/server.go` — route, above the bare `/hosts/` catch-all +- `hub/internal/web/templates/host_detail_body.html` — Console access card + fetch-on-demand script +- `hub/CHANGELOG.md`, `REUSE.md`, `CONTEXT.md` (ruling **S-4**), + `documentation/runbooks/break-glass.md` (§3.1 + §5), + `documentation/architecture/00-capability-map.md` (L120), + `documentation/backlog/ROADMAP.md` + `documentation/backlog/OPEN-ITEMS.md` (**R-133**) -`RoleForStorage` types a `local-dir` on a non-system device as **user-data**, so the pre-existing -eject role gate PASSED the drive E-1 had just made the vzdump target. `POST /disks/eject` on -`/mnt/nvme-1tb` or `/mnt/hdd_1` would have **succeeded silently**, taking the only local whole-guest -backup with it while the box kept reporting a configured tier. +## 3. Commits pushed to `main` -Fixed in agent v0.111.0/0.112.0: `handleDiskEject` and `handleDiskDecommission` refuse `409` when the -mount backs a configured tier, naming the storage and the remedy — the op is *ordered*, not -forbidden. +*(filled in at push — see `git log --oneline`)* -**Deliberately NOT a role reclassification.** Making `RoleForStorage` return `RoleBackup` would -refuse every legitimate eject of the customer's own data drive, because on both boxes that drive -**is** the target. `TestEjectStillAllowedOnANonTargetDrive` pins that, and the over-correction is -red-proofed. +## 4. Test results + red-proofs -**Live on both boxes:** +Every test drives `RequireAuth(ServeHTTP)` — never a handler function directly. -``` -demo-hp felhom-backup mount=/mnt/nvme-1tb role=user-data backup_target=True -demo-felhom felhom-backup mount=/mnt/hdd_1 role=user-data backup_target=True -``` +| Test | Scenario | Result | +|---|---|---| +| `TestReveal_A_PageNeverCarriesTheSecret` | A — render, canary absent from the WHOLE body | PASS | +| `TestReveal_B_RevealDeliversAndAudits` | B — 200 + no-store + payload + exactly 1 event + no log leak + 0 notifications | PASS | +| `TestReveal_C_NotVaulted` | C — explanatory card, no control, 404, zero events | PASS | +| `TestReveal_D_CSRFRequired` | D — 403, no leak, zero events; + the with-token discriminator | PASS | +| `TestReveal_E_MethodGateAndRouteOrder` | E — 405 and no fall-through to the host page | PASS | +| `TestReveal_F_UnknownHost` | F — 404, no panic | PASS | +| `TestReveal_G_Unauthenticated` | G — 401, no leak, zero events | PASS | +| `TestReveal_UnboundHostRevealsWithoutAnEvent` | §8 edge — 200, no event, log is the record | PASS | +| `TestGetHostRecoveryMeta_MetadataOnly` | H — store round-trip + absent cases | PASS | -That single line is the exposure: `role=user-data` (which the gate permits) yet `backup_target=True`. -Both ejects returned **409** with the full message; both decommissions **409**; both drives still -mounted with `dump/` intact — the refusal had no side effect. `/var/lib/vz` is still refused by the -**pre-existing** role gate (`role: system`), proving the new gate is narrow. +**Red-proofs — each mutation applied, observed failing, reverted:** -## 3. E-2b — the alarm that was defined and never called +| # | Mutation | Observed | Reverted | +|---|---|---|---| +| A | `hostDetailData` gains `RecoverySecret` (from `GetHostRecoveryCredential`) + `data-secret="{{.RecoverySecret}}"` on the card | `FAIL … SECRET LEAK: the vaulted console password appears in the rendered host page` | yes | +| B | delete the `SaveEvent` block in `handleHostRevealRecoveryCredential` | `FAIL … recovery_credential_revealed rows = 0, want exactly 1` | yes | +| D | short-circuit the `ServeHTTP` CSRF check (`if false && !s.validateCSRF(r)`) | `FAIL … reveal without CSRF = 200, want 403` | yes | +| E | move the new `case` BELOW `case strings.HasPrefix(path, "/hosts/")` | `FAIL … GET on the reveal route = 404, want 405` | yes | -`NotifyStorageDisconnected` and `NotifyStorageReconnected` were registered in `allowedEventTypes`, in -`DefaultEnabledEvents`, and given Hungarian hub copy — and **called from nowhere**. A drive going -absent produced stopped apps, a `[WARN]` log and a UI badge, then silence on every channel. A drive -that is *only* a backup target has no apps to stop, so it was silent twice over. Fifth instance of -this class; found by Phase 0, not by a failure. +**Red-proof A did not land on the first attempt, and that is worth recording.** The first mutation +edited only the template; a `{{.RecoverySecret}}` against a map with no such key renders empty, so +the test passed and would have certified nothing. The proof landed only once the *data* half was +mutated too — i.e. the assertion is pinned to the view-model, not to template text. -Now wired in `ReconcileDriveGates`, with the target case raising the more specific -`backup_target_absent` (error) and recovering as `backup_target_restored` (info — the pairing-gated -pattern; `severityNotifies` NOT widened). Never both: two mails for one event trains people to -ignore the channel. +**Red-proof E's symptom differed from the prediction.** The spec expected the misordered route to +render the host page (200); it actually 404s, because the catch-all takes +`demo-felhom-8363b5/reveal-recovery-credential` as the host id and `GetHost` misses. The test goes +red either way, and its second assertion (no `Console access` in the body) still pins the +fall-through case. -## 4. The bug I introduced and caught before deploy +**One spec assertion caught a real gap during development.** Scenario A's requirement that the page +"contains a Reveal control targeting `/hosts/{id}/reveal-recovery-credential`" failed at first: the +URL was assembled in JS (`'/hosts/' + encodeURIComponent(hostID) + '/…'`) and appeared nowhere in the +DOM. Fixed by putting the endpoint in `data-reveal-url` on the button **and making the fetch read it +from there**, so the string the render test asserts is the string the request uses — an attribute +nothing reads would have been a hollow assertion. -`ReconcileDriveGates` resolves the target as `isTarget[a.Path]`, and **`a.Path` is the registered -GUEST path** (`/mnt/felhom-drives/`), not the agent's host `MountPath`. I had keyed the map on -`MountPath` alone, so the backup-target branch was **unreachable** — every absent drive, the target -included, would have fallen through to the generic event. +## 5. Test count -It would have looked wired, passed its own unit tests, and shipped silently wrong on exactly the -drive it exists for. Caught by tracing `a.Path` back to its source instead of assuming. Fixed in -v0.184.1 (both keys, mirroring `planDriveGates`), red-proofed, and the **0.184.0 image is superseded -and was never deployed**. +**550 → 559** (8 web + 1 store). Full suite `go build ./... && go vet ./... && go test ./...` in +`hub/`: **rc=0, all 17 packages ok**. Run as a separate command from the commit, per standing rule 1. -## 5. Tests and red-proofs +## 6. Deployed version -| Red-proof | Observed failure | -|---|---| -| hub allowlist entry removed | `backup_target_absent must be in allowedEventTypes, or POST /event 400s…` | -| hub Hungarian copy shortened | `must state the consequence…, got: "Meghajtó hiányzik."` | -| C — auto-elevation on register | `registering drives assigned the backup target "/mnt/hdd_1"` | -| E-2c — eject guard removed | `eject of the backup-target drive SUCCEEDED (200)` | -| E-2c over-correction | the gate blocked `/mnt/spare` (a non-target) | -| keying reverted to MountPath | `not resolvable by its GUEST path — …the branch would never fire` | +*(filled in after the build/manifest/sync steps)* -Every mutation was **verified to have landed** before the result was trusted. The first attempt -silently no-op'd (gofmt had realigned the map) and the test "passed" — a false proof. +## 7. NOT yet live-validated — awaiting the operator -Controller notify tests observe the **wire** (a real `Notifier` against an `httptest` hub), not a -mock, because the failure class is "nothing arrives". +**That the revealed password actually authenticates at `https://:8006` as `root@pam` on +`demo-felhom-8363b5`.** This is the only test that proves the hub's copy still matches the box, and +it needs a browser and a real login — CC has neither here (no `claude-in-chrome` on DooPlex). Every +other leg is endpoint-level validated (§6). -## 6. What is NOT done +## 8. Teardown -- **Part 2** (installer Case A/B), **Part 3** (the offer + the agent-side move), **Part 4** (the - degraded Hungarian banner). Phase 0 inverted their emphasis: the installer has **no drive-enrollment - step**, so at install the common case is system-drive-only and **Part 3 is the primary path**. -- **E-2a** — the guarded wrapper. The agent holds neither `Datastore.Allocate` at `/storage` nor - `Permissions.Modify`, and its sudoers has no `pvesm`/`pveum`. Use the wrapper pattern; **do not - widen the agent's PVE role.** -- **`backup_target_absent` is not proven END-TO-END live.** Its components are: the controller emits - the right type/severity (proven at the wire), the hub allowlists it (unit-pinned, image deployed), - and the keying is pinned. Proving the whole chain needs a live drive to actually go absent, which - means unmounting an enrolled drive carrying customer data and the guest bind on a remote box — - the same thing declined in E-1, for the same reason. -- Full drive-loss recovery (physical access) — carried forward from E-1. +**This run provisioned nothing** — no guest, no VM, no customer, no drive, no external resource. No +teardown obligation. -## 7. Observations +## 9. Backlog rows opened / closed / re-ranked -- **`peti-felhom` untouched.** Its whole-guest backup still shares a device with its guest, so a - drive failure there is offsite-only recovery. Accepted until the operator's reinstall; re-evaluate - if that slips past ~2026-09-01. -- The fleet was briefly split (demo-hp updated first). Left that way it would have protected one of - two identically-exposed boxes, so demo-felhom was rolled to the same proven artifacts. -- `StorageTarget.Role` carries a documented vocabulary including `vzdump-target` and is always `""`. - Checked — it is a deliberate unpopulated field ("hub-owned, slice 10"), **not** a sixth dead seam. +- **Opened: `R-133`** — the vaulted secret is plaintext at rest, so every hub DB backup is a + fleet-wide console-credential dump; envelope-encrypt `host_recovery.secret` under a KEK held + outside the DB. Added to `ROADMAP.md` and `OPEN-ITEMS.md` (owner CC, `READY (M)`). Named the + capability-map row it would flip. +- **Closed / re-ranked: none.** +- **ID collision, resolved:** the spec predicted R-128. The concurrent session's *uncommitted* WIP in + `OPEN-ITEMS.md` had already taken **R-128 through R-132**, so this item took **R-133**. An ID + register that lives in a file two sessions edit at once cannot allocate safely by reading committed + state — worth noting, not fixed here. ---- +## 10. Observations — recorded, deliberately NOT acted on -# E-2 increment 2 (same day): the wrapper, the installer, the offer and the banner - -**Live fleet:** hub **0.81.0**, agent **0.113.0**, controller **0.185.1**, host-install **1.22.0** -(script; no reinstall performed). Both demo boxes. `peti-felhom` untouched. - -## E-2a — the guarded wrapper - -`configs/felhom-backup-target-apply` behind a literal `FELHOM_BACKUPTARGET` sudoers alias. **The -agent's PVE role was not widened** — it still holds neither `Datastore.Allocate` at `/storage` nor -`Permissions.Modify`. - -All five laws proven live as root on demo-hp, with **0 stray storages created**: - -| Probe | Result | -|---|---| -| F-1 subdirectory (`/mnt/nvme-1tb/dump`) | REFUSED — "must be the drive's OWN mountpoint" | -| F-2 unmounted path | REFUSED — "would silently retarget onto the system drive" | -| root device (`/`) | REFUSED | -| idempotent re-apply on the real target | no-op, exit 0 | -| repointing an existing id | REFUSED | - -`POST /backup/target` drives it **create → grant → config**, and deliberately does **not** restart the -agent: it returns `restart_required` so the restart happens behind the caller's own immediate -in-flight check. That is E-1's spurious-tier-failure mistake encoded as a contract, and a test asserts -the handler never restarts. - -## Part 2 — the installer (v1.22.0) - -Case A creates the target through the same wrapper; Case B records DEGRADED in two warning lines and -**still succeeds** — a single-drive appliance is valid. Scenario F returns early with **0 wrapper -calls**; the red-proof (guard removed) shows **2**, i.e. it would have "corrected" a correct box. - -**Installer-logic-tested, NOT install-tested** — the harness runs the extracted functions with -stubbed `pvesm`/wrapper. No reinstall was performed. - -## Parts 3+4 — the offer and the banner - -`POST /api/storage/backup-target/assign` is the **only** writer of the role. `GET -/api/storage/backup-target` drives the banner. Hungarian copy is FACT → CONSEQUENCE → REMEDY. - -**Scenario E proven live on both boxes** — healthy renders nothing: - -``` -demo-hp {"degraded":false,"known":true,"label":"NVME 1TB","target":"felhom-backup"} -demo-felhom {"degraded":false,"known":true,"label":"USB HDD 1TB","target":"felhom-backup"} -``` - -No `message` key at all. Red-proofed: adding a reassurance to the healthy branch fails Scenario E. - -## Three defects I introduced and caught - -1. **The offer endpoints were unreachable.** Registered as `/api/backup-target` inside - `ServeStorageAPI`, which `main.go` mounts only at `/api/storage/`. Every unit test passed because - they called the handlers directly; the **first live call** returned `endpoint not found`. Fixed in - v0.185.1 with a test that asserts the dispatcher's own source contains both routes. -2. **A hollow test.** `TestUnknownStateRendersNothing` used `{Known:false}` with `Degraded` left - false, so it passed even with the `!Known` guard deleted. Its own red-proof exposed it; the - fixture is now `{Known:false, Degraded:true}`. -3. **A gofmt-realignment no-op** (again): a `sed`-style replace silently matched nothing because the - formatter had changed the spacing. Every mutation is now verified to have landed before its result - is trusted. - -## Still not live-proven - -- The **DEGRADED banner** and the **offer acceptance** — both boxes are healthy, so neither state - occurs naturally. Unit- and red-proofed only. -- **`backup_target_absent` end-to-end** — needs a live enrolled drive to go absent. -- **Case A/B on a real install** — installer-logic-tested only. -- **Full drive-loss recovery** — physical access. +1. **`customer_unified.html`'s `data-secret` / `toggleSecret` widget embeds the plaintext in the page + HTML on every load.** It therefore lives in the back/forward cache, in "save page as", and in any + DOM-capturing screenshot. Acceptable for one customer's retrieval passphrase; it is the reason + this task built fetch-on-demand instead of reusing it. Not refactored — out of scope. +2. **`validateCSRF` returns `true` when no session cookie is present** (`server.go`, the Basic-Auth + path). So a Basic-Auth caller reaches the reveal endpoint **without any CSRF token**. That is the + pre-existing hub-wide contract, not something this endpoint introduced, and it is what makes the + §6 curl validation possible at all — but it does mean "CSRF-gated" is true only for session + callers. Recorded, not changed. +3. **The Console access card also renders on the customer page's Host tab**, because + `host_detail_body` is shared and `configs.go` builds its view models through the same + `hostDetailData`. That is per-host and operator-only (the hub has no customer login), so it does + not breach the "never on the hosts LIST" rule — but it is a second surface, and it is stated here + rather than left to be discovered. +4. **A concurrent session shares this clone.** Its in-flight edits to `OPEN-ITEMS.md`, + `RUNBOOK-manual-build.md`, `RUNBOOK-publish-0.79-0.110-2026-07-10.md` and two new untracked files + were **not** swept into this commit: every path was staged explicitly, and `OPEN-ITEMS.md` was + staged as a hand-built blob (`HEAD` + the R-133 row only) so the foreign WIP stayed unstaged in + the working tree. `REPORT.md` was taken here because that session had already chosen the + `REPORT-tester-gate-2026-07-31.md` sibling. diff --git a/REUSE.md b/REUSE.md index 5ef2264..ec19d0e 100644 --- a/REUSE.md +++ b/REUSE.md @@ -71,6 +71,7 @@ | Symbol | File | Short signature | Use for | Gotchas | |---|---|---|---|---| | `(*Server).hostDetailData` | hub/internal/web/hosts.go (~L282) | `(host *store.Host, r) map[string]interface{}` | The ONE view-model builder for the shared `host_detail_body` sub-template (standalone `/hosts/{id}` + customer Host tab) | Booleans/counts only for DR/escrow; carries `Deletable` (= status != "ok") which gates the danger-zone card. Never add a secret field. | +| `(*Store).GetHostRecoveryMeta` + `(*Server).handleHostRevealRecoveryCredential` | hub/internal/store/host_recovery.go · hub/internal/web/hosts.go | `(hostID) (*HostRecoveryMeta, error)` · `POST /hosts/{id}/reveal-recovery-credential` | The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) | **Use `GetHostRecoveryMeta` on any page-render path** — its struct and its `SELECT` both omit the `secret` column, so it cannot leak one; `GetHostRecoveryCredential` (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE `recovery_credential_revealed` event via `SaveEvent` and calls NO dispatcher (the `handleRequestLogTail` shape). `api/handler.go handleAdminGetRecoveryCredential` (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133. | | `host_detail_body` sub-template | hub/internal/web/templates/host_detail_body.html | `{{template "host_detail_body" .}}` | Rendering a host's detail sections on ANY surface | One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with `{{.HostID}}` (the customer page renders N instances). | | `(*Store).ListHostsByCustomer` | hub/internal/store/store.go (~L1620) | `(customerID) ([]Host, error)` | A customer's hosts, host_id order | A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer. | | `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). | diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index d5281f0..c3fce95 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -117,7 +117,7 @@ | Forgot dashboard password → instant reset code | controller v0.123, hub | **PROVEN-LIVE** | `DRILL-day0-take2-2026-07-12` F-15 (live re-run of the exact failure path: hash applied 1s after request, code accepted first try) | | | Multiple household users / per-person accounts | — | **MISSING** | — | Single dashboard password; acceptable for alpha → R-15 | | WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | `SPIKE-oob-wg-operator-peer-2026-07-05`, `SPIKE-felhom-sshd-2026-07-05` | Mutual-repair desired-state arc not built → R-13 | -| Break-glass management-plane recovery | agent v0.71, hub v0.34 | **IMPLEMENTED** | `runbooks/break-glass.md` | | +| Break-glass management-plane recovery | agent v0.71, hub v0.84 | **IMPLEMENTED** | `runbooks/break-glass.md` | hub v0.84.0 adds an **operator-SESSION** retrieval path (host page → Console access → Reveal; `POST /hosts/{id}/reveal-recovery-credential`, CSRF-gated, writes a customer-visible `recovery_credential_revealed` event) beside the pre-existing **global-key** one (`GET /api/v1/admin/hosts/{id}/recovery-credential`), which is untouched and stays the route for when the hub UI itself is down. Still IMPLEMENTED, not PROVEN-LIVE: the UI path has not been exercised on a real lockout, and that the revealed password authenticates at `:8006` is operator-verified only. The vaulted secret is plaintext at rest → **R-133** | ## F. Notifications & monitoring diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 0082b83..6f15906 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -70,6 +70,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC | | **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC | | **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC | +| **R-133** | **The vaulted break-glass console credential is PLAINTEXT AT REST — every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim, so any copy of the SQLite DB (Longhorn snapshot, PBS backup of the hub PVC, a hand-taken copy during a diagnosis) carries root console access to every Felhom host in one file | **READY (M) — NEW 2026-07-31** | — | **The deferred leg of hub v0.84.0** (Console access card), filed separately because v0.84.0 changed only WHO can retrieve the secret, never how it is stored. v0.84.0 makes it more worth doing, not more broken: retrieval now rides the hub SESSION, so the DB and the login password are jointly the whole protection (ruling **S-4**, `CONTEXT.md`). Fix shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held outside the DB** — the hub already proves it can hold something it cannot itself read (escrow blobs), and that contrast is the argument. Two constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the whole point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Would flip the capability-map row **"Break-glass management-plane recovery"**, which today reads IMPLEMENTED with this as its caveat | CC | ## Why the TOP READY rows rank this way diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 41c8b0c..18c8e60 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -110,6 +110,7 @@ | R-107 | **No offsite action unpacks the named-volume tars Tier-3 captures on every run.** `ReconstituteFromOffsite` skips the unit outright (`internal/backup/offbox_reconstitute.go:284-289`); `PlaceOffsiteRestore` places it only when the live unit is ABSENT (`internal/backup/offbox_restore.go:352-356`). The DB **is** replayed, from the scratch unit's `db-dumps/` | M | READY — 2026-07-28 | 52 of 53 catalog templates keep data in Docker named volumes, so for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists (full restore → place → Tier-1 restore, which does unpack tars) but **no single action does it and no UI routes it** — the R-103 shape one tier over. Sibling of R-102: both are "captured and never read". Flips: the offsite-restore row in map §C; `07` §6.3, §7.2 | | R-108 | ~~Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5~~ | M | **SHIPPED** — controller v0.187.0, 2026-07-30 | **D5 UNBLOCKED.** One fail-closed predicate (`Settings.RefuseAsAppNamespace`) consulted by every placement surface, so no `backups/` tree can exist inside the share-root bind. The bind was deliberately NOT narrowed — it is load-bearing (`:rslave` automount wake, 2026-07-22 probe) and unscopable (apps on a share store at `/`; creating a `userdata/` layer would write Felhom convention onto a customer's NAS). **Five surfaces, not the four this row named** — decommission-with-migrate guarded only its SOURCE; and the boundary is the deploy POST, not the dropdown. Zero apps on network storage fleet-wide, so nothing stranded. The `.fab`-onto-NAS "second effect" is split out as **R-126**. Flipped: `07` §7.3 (D5 → unblocked), §10.1 (CLOSED), `00-capability-map.md` §D (new PROVEN-LIVE row). `audits/R108-network-app-namespace-2026-07-30.md` | | R-127 | **`data_key: true` is unreliable (4+ encryption keys unflagged, contradicting the catalog's own labels), and O4 can regenerate a DB password that no longer matches the restored data directory** | S/M | READY — NEW 2026-07-30 | Found by **D5's Part 0**, and the reason D5's boundary became `type: secret` rather than `data_key`. Leg (a): flag the missing keys (catalog-only) + pin flag-vs-label agreement; the residual risk after D5 is that the **fail-closed gate** keys on `data_key`, so an unflagged key missing from both sources lets the restore proceed onto undecryptable data. Leg (b): a regenerated DB password is silently wrong — `POSTGRES_PASSWORD` is ignored once PGDATA is non-empty, so the app cannot authenticate while the replay still succeeds over the local trust socket (proven live on `postgres:16-alpine`). v0.188.0 corrected the false "stored data is unaffected" WARN but added no guard. Flips: `07` §7.4 | +| R-133 | **The vaulted break-glass console credential is PLAINTEXT AT REST, so every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim (`hub/internal/store/host_recovery.go` — *"a hub-held secret, operator-retrievable (NOT zero-knowledge like escrow)"*), so anything that copies the SQLite DB — a Longhorn snapshot, a PBS backup of the hub PVC, a hand-taken copy during a diagnosis — carries root console access to every Felhom host in one file, with no second factor and no key to withhold | M | READY — NEW 2026-07-31 | **The DEFERRED LEG of hub v0.84.0**, filed as its own ID because v0.84.0 changed only WHO can retrieve the secret, never how it is stored — the at-rest shape predates it and is untouched by it. **v0.84.0 made it more worth doing, not more broken:** putting retrieval behind the hub session means the hub login password alone now unlocks console root fleet-wide, so the DB and the login are the whole of the protection. Shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held OUTSIDE the DB** (k8s Secret / out-of-band file, the way `manifests/` already keeps the bearer out of git), so a DB copy is opaque the way escrow blobs already are — the contrast is the argument, since the hub already proves it can hold a secret it cannot itself read. Constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the entire point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Flips: `architecture/00-capability-map.md` **"Break-glass management-plane recovery"** — the row that today reads IMPLEMENTED with a plaintext-at-rest caveat | | D5 | ~~**Move app secrets into the LOCAL recovery unit** so Tier-1/Tier-2 restore stop needing the guest and stop needing R~~ | M | **SHIPPED + PROVEN-LIVE** — controller v0.188.0, 2026-07-30 | **The arc's architectural centrepiece. Tier-1/2 no longer depend on the whole-guest tier — a customer needs the DRIVE AND NOTHING ELSE.** Part 0 tested this row's own premise and **rejected** it: data-keys-only is both insufficient and unsafe, because `data_key` is unreliable (→ **R-127**) and a DB password is not resettable in practice (`POSTGRES_PASSWORD` is ignored once PGDATA is non-empty, so a regenerated value leaves the app unable to reach its own restored rows while the dump replay still reports success — proven on `postgres:16-alpine`). **Operator ruling: `type: secret` travels (45 fields), `type: password` never (7) plus a code register (`vaultwarden/ADMIN_TOKEN`); plaintext, because withholding the internet-reachable class is what licenses it — the two are coupled.** `stacks.PortableSecretEnvVars` is the single boundary; the register is code, not a catalog flag (R-97a). **Precedence: the UNIT WINS** (its secrets match the data being restored, not merely the newest), pinned both directions. Fail-closed data-key gate UNCHANGED. Manifest schema 2; schema-1 units still restore. Proven live on a scratch drill guest: AdventureLog restored with the guest `app.yaml` moved aside (`secrets recovered=2/2`, 27.6 s) and **the app read the seeded row over TCP with its own credential**; Grafana's admin password withheld with **0 hits** across the backup namespace. 4 red-proofs each verified to land. `audits/D5-drive-alone-restore-2026-07-30.md` Flips `07` §3/§7.1/§7.3/§7.4/§8/§10.1 + a new capability-map row | | R-126 | **A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS.** `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | S | READY — 2026-07-30 | Split out of R-108 on its closure. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (`07` §7.3 records the reasoning). Fix = filter network paths from the export destination list, or force the bundle password when the destination is a share. Flips: `07` §5 | | R-109 | ~~The DR recipe records no backup target~~ | XS | **SHIPPED** — agent v0.118.1 + hub v0.83.0, 2026-07-30 | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` (the scheduler's own function) + the mountpoint that separates `/mnt/hdd_1` from `/var/lib/vz`; unresolvable is recorded as unresolvable, never defaulted. Required an unscoped HUB half: `AssembleDRRecipe` allow-lists top-level keys → **R-122**, a fourth recipe-completeness defect that had already shipped (`offsite_restic` dropped for the feature's whole life). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 | diff --git a/documentation/runbooks/break-glass.md b/documentation/runbooks/break-glass.md index 1b65589..b77698b 100644 --- a/documentation/runbooks/break-glass.md +++ b/documentation/runbooks/break-glass.md @@ -39,13 +39,26 @@ Layer 2 heals within ~60s with nobody logged in. If you just saw the KEXINIT-res retry SSH. If it's back, check the hub for a `mgmt_plane_healed` warning and go to §4 (find the cause). ### 3.1 If SSH is still dead — reach the PVE web console (break-glass) -1. **Retrieve the vaulted root@pam password** (operator/global key): +1. **Retrieve the vaulted root@pam password.** + + **Primary route — the hub UI (hub v0.84.0):** hub → **Hosts** → the host → the **Console access** + card → **Reveal**. The password shows for 60 s with a Copy button. The card also states which user + it is for and when it was last vaulted, so a stale copy is visible before you try it. A host with + nothing vaulted says so and says why, instead of showing a blank. Needs only the hub login + password — no out-of-band key. + + **Fallback — the global-key curl. Use this when the hub UI itself is unavailable** (that is the + whole reason it still exists, and why it deliberately has no dependency on the hub's session + layer): ``` curl -s -H "Authorization: Bearer " \ https://hub.felhom.eu/api/v1/admin/hosts//recovery-credential # → {"host_id":"…","username":"root@pam","password":"…","set_at":"…"} ``` (A per-host key is refused here by design — only the operator key reads a console password back.) + + Either route is an **operator action against a customer's box**, and the UI route records it: see + §5. 2. **Open the PVE web UI** `https://:8006`, log in `root@pam` + that password, realm *Linux PAM standard authentication*. 3. **Node → Shell** (xterm.js root console). Or navigate directly: @@ -87,6 +100,17 @@ Fix the cause; the three layers are the safety net, not the cure. wipes `state.json`, so step 4b re-randomizes — re-run the set-and-vault afterward. Background: on each full reinstall this is exactly why `root@pam` "kept changing" until it was pinned. - **The password is never logged or filed** — it exists only in the hub vault (retrieve as in §3.1). +- **Revealing it through the hub UI writes a customer-visible event line** (hub v0.84.0): + `recovery_credential_revealed`, info severity, on that host's customer timeline, in Hungarian — + *„Az üzemeltető lekérte a géped konzolos hozzáférési jelszavát (távoli hibaelhárítás)."* Nobody is + emailed (no dispatcher call), and the register records **accesses**, so two reveals write two + lines. Deliberate, for a product sold on data sovereignty; it mirrors how a remote log pull is + already recorded. An **unbound** host has no customer to tell — there the hub `[INFO]` log line is + the only record. The global-key curl path writes no event. +- **The hub login password now unlocks console root on every managed box.** Before v0.84.0, + retrieval also required the global API key. Accepted for a single-operator, HU-geo-fenced hub that + already holds these passwords in plaintext at rest — a convenience trade, and a real one. The + plaintext-at-rest half (every hub DB backup is a fleet-wide console-credential dump) is **R-133**. - **Least-privilege console user** instead of root@pam was probed as a non-blocking observation: PVE has historically gated the node Shell to `root@pam`, so G1 ships the proven root@pam path. Revisit if PVE later allows a scoped realm user to open a node Shell. diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index f434a93..a67564a 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,58 @@ +## v0.84.0 — Break-glass console credential on the host page (2026-07-31) + +**The credential existed and was not reachable when it was wanted.** Every Felhom-installed box has +had a strong random `root@pam` console password since TASK G1 — set on the box by +`felhom-host-install.sh` step 4b, vaulted in the hub at day 0, live for three hosts today, and used +for real during the sshd incident. The only way to read it back was a hand-written `curl` against +`/api/v1/admin/hosts//recovery-credential` carrying the **global operator key** — a different +secret from the hub login password, kept out-of-band. In practice the PVE web console on a demo box +felt locked. + +**The host page grows a `Console access` card.** By default it states only that a credential is +vaulted, for which user, and when it was last set. A **Reveal** button fetches the plaintext on +demand and shows it for 60 s with a Copy button; masking clears the JS variable, and the mask also +fires on a second click and on `visibilitychange → hidden`. A host with nothing vaulted says so, and +says why (byo host, or step 4b never ran), with no Reveal control at all. + +**The secret is never rendered into the page — that constraint shapes the whole change.** The render +path calls a new `store.GetHostRecoveryMeta`, whose struct and whose `SELECT` both omit the `secret` +column, so it is *structurally* incapable of carrying it; `hostDetailData` gains exactly three keys +(`RecoveryVaulted`, `RecoveryUsername`, `RecoverySetAt`). The plaintext crosses the wire only in the +response to `POST /hosts/{id}/reveal-recovery-credential` — `Cache-Control: no-store`, CSRF-gated at +the `ServeHTTP` level, POST precisely so that gate applies and so a secret is never retrievable by +URL alone (prefetch, history, referrer). The load-bearing test asserts the canary appears **nowhere** +in the rendered response — attribute, comment, inline script or JSON blob. + +**Deliberately NOT the `customer_unified.html` `data-secret` widget**, which embeds the plaintext in +the page HTML on every load: acceptable for one customer's retrieval passphrase, not for console root +on every box in the fleet (it survives in the bfcache, in "save page as", and in any DOM-capturing +screenshot). That widget is untouched and recorded as an observation. + +**Transparency, matching the log-pull precedent.** A delivered reveal writes one +`recovery_credential_revealed` event (info, source `hub`, Hungarian) on the host's customer timeline +— `SaveEvent` alone, no dispatcher call, so nobody is emailed. Two reveals write two events: the +register records **accesses**, not states. A 404 is not an access and writes nothing. An unbound host +reveals fine and writes no event (no customer to tell); the `[INFO]` hub line is then the only +record, and it carries the username and a length — never the password. + +**The global-key API path is untouched, by design.** `handleAdminGetRecoveryCredential` is the +break-glass route for when the hub *UI* is the thing that is broken; coupling it to the session layer +would remove exactly the independence that makes it a fallback. + +**Recorded as a real trade, not a free one:** the hub session password alone now unlocks console root +on every managed box, where retrieval previously also required the global API key. Accepted for a +single-operator, HU-geo-fenced hub that already stores these passwords in plaintext at rest — and the +plaintext-at-rest half is now filed as **R-133** (envelope-encrypt `host_recovery.secret` under a KEK +held outside the DB, so a hub DB backup stops being a fleet-wide console-credential dump). + +Files: `hub/internal/store/host_recovery.go` (+`GetHostRecoveryMeta`, `HostRecoveryMeta`), +`hub/internal/web/hosts.go` (+`handleHostRevealRecoveryCredential`, `hostDetailData`), +`hub/internal/web/server.go` (route, **above** the bare `/hosts/` catch-all), +`hub/internal/web/templates/host_detail_body.html` (card + fetch-on-demand script). +Tests 550 → 559; four red-proofs (A page-leak, B audit event, D CSRF gate, E route order) each run, +observed failing, and reverted. The E proof is a seam test driving `ServeHTTP`: a handler-level test +cannot see that defect, because the handler is correct and simply never runs. + ## v0.83.0 — R-109 + R-122: the recipe assembly stops dropping sections (2026-07-30) Pairs with **agent v0.118.0** (R-106 + R-109). The agent half is useless without this one. diff --git a/hub/internal/store/host_recovery.go b/hub/internal/store/host_recovery.go index 0a5357c..a1184a7 100644 --- a/hub/internal/store/host_recovery.go +++ b/hub/internal/store/host_recovery.go @@ -47,6 +47,34 @@ func (s *Store) GetHostRecoveryCredential(hostID string) (*HostRecoveryCredentia return &c, nil } +// HostRecoveryMeta is the NON-SECRET shape of a vaulted break-glass credential: what the operator's +// host page shows without the plaintext ever entering the rendered document. The secret column is +// deliberately absent from both the struct and the query — the render path must be unable to carry it. +type HostRecoveryMeta struct { + HostID string + Username string + SetAt time.Time +} + +// GetHostRecoveryMeta returns a host's credential metadata, or (nil, nil) if none is vaulted. +// Use this — NOT GetHostRecoveryCredential — on any path that renders a page: the secret can only +// leave the hub through the explicit, CSRF-gated, audited reveal endpoint. +func (s *Store) GetHostRecoveryMeta(hostID string) (*HostRecoveryMeta, error) { + var m HostRecoveryMeta + var setAt string + err := s.db.QueryRow( + `SELECT host_id, username, set_at FROM host_recovery WHERE host_id = ?`, hostID). + Scan(&m.HostID, &m.Username, &setAt) + if err == sql.ErrNoRows { + return nil, nil + } + if err != nil { + return nil, err + } + m.SetAt = parseSQLiteTime(setAt) + return &m, nil +} + // HasHostRecoveryCredential reports whether a host already has a vaulted credential (day-0 idempotency: // don't regenerate/re-set on a re-run unless --rotate). func (s *Store) HasHostRecoveryCredential(hostID string) (bool, error) { diff --git a/hub/internal/store/host_recovery_test.go b/hub/internal/store/host_recovery_test.go index 2ec4bc4..8e6230f 100644 --- a/hub/internal/store/host_recovery_test.go +++ b/hub/internal/store/host_recovery_test.go @@ -43,6 +43,43 @@ func TestHostRecoveryCredential_RoundTripUpsertAndAbsent(t *testing.T) { } } +// GetHostRecoveryMeta is the render path's accessor (hub v0.84.0). It returns username + set_at +// and, by CONSTRUCTION, cannot return the secret: neither HostRecoveryMeta nor the SELECT names the +// `secret` column, so there is no runtime assertion to write for that half — adding a Secret field +// would not fail this test, it would fail to compile at every call site that never asked for one. +// The runtime half asserted here is the metadata round-trip and the absent case. +func TestGetHostRecoveryMeta_MetadataOnly(t *testing.T) { + s := newTestStore(t) + if err := s.UpsertHost(&Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil { + t.Fatalf("UpsertHost: %v", err) + } + + // absent → (nil, nil), never an error the page has to special-case + m, err := s.GetHostRecoveryMeta("h1") + if err != nil || m != nil { + t.Fatalf("absent meta: got %+v / %v (want nil,nil)", m, err) + } + + if err := s.SaveHostRecoveryCredential("h1", "root@pam", "s3cret-Aa1"); err != nil { + t.Fatalf("SaveHostRecoveryCredential: %v", err) + } + m, err = s.GetHostRecoveryMeta("h1") + if err != nil || m == nil { + t.Fatalf("GetHostRecoveryMeta: %+v / %v", m, err) + } + if m.HostID != "h1" || m.Username != "root@pam" { + t.Fatalf("meta mismatch: %+v", m) + } + if m.SetAt.IsZero() { + t.Fatal("SetAt did not parse — the card cannot render staleness without it") + } + + // an unknown host is the absent case too, not an error + if m, err := s.GetHostRecoveryMeta("nope"); err != nil || m != nil { + t.Fatalf("unknown host: got %+v / %v (want nil,nil)", m, err) + } +} + func TestGetHostMgmtPlaneStates_ParsesHealMarker(t *testing.T) { s := newTestStore(t) if err := s.UpsertHost(&Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil { diff --git a/hub/internal/web/hosts.go b/hub/internal/web/hosts.go index c8e8eb8..5f9c95e 100644 --- a/hub/internal/web/hosts.go +++ b/hub/internal/web/hosts.go @@ -438,6 +438,14 @@ func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]in drBundle, _ := s.store.GetHostDRBundle(host.HostID) escrow, _ := s.store.GetHostEscrow(host.HostID) + // v0.84.0 Console access — presence + username + set_at ONLY. GetHostRecoveryMeta cannot carry + // the secret (its query does not select the column); the plaintext reaches the operator solely + // through POST /hosts/{id}/reveal-recovery-credential. + recoveryMeta, err := s.store.GetHostRecoveryMeta(host.HostID) + if err != nil { + s.logger.Printf("[ERROR] host recovery meta %s: %v", host.HostID, err) + } + return map[string]interface{}{ "WrapperDrift": wrapperDrift, "ReportedWrapperSHA": reportedWrapperSHA, @@ -467,6 +475,20 @@ func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]in // v0.60.0 Part B: retained superseded escrow blobs (data-first — old passphrases stay // R-recoverable). Operator-only surface. "SupersededEscrowCount": func() int { n, _ := s.store.CountSupersededEscrow(host.HostID); return n }(), + // v0.84.0 break-glass Console access card. NEVER add a key holding the secret. + "RecoveryVaulted": recoveryMeta != nil, + "RecoveryUsername": func() string { + if recoveryMeta != nil { + return recoveryMeta.Username + } + return "" + }(), + "RecoverySetAt": func() time.Time { + if recoveryMeta != nil { + return recoveryMeta.SetAt + } + return time.Time{} + }(), // v0.46.0 Diagnostics: pending log pulls + received/blocked bundles (72 h TTL). "LogBundles": s.hostLogBundleRows(host), "CSRFToken": s.getCSRFToken(r), @@ -511,6 +533,61 @@ func (s *Server) handleHostDeleteImpact(w http.ResponseWriter, r *http.Request, }) } +// handleHostRevealRecoveryCredential — POST /hosts/{id}/reveal-recovery-credential (v0.84.0). +// The operator-SESSION counterpart to the global-key API path (api/handler.go +// handleAdminGetRecoveryCredential), which stays untouched and remains the break-glass route for +// when this UI is itself unavailable — coupling it to the session layer would remove exactly the +// independence that makes it a fallback. +// +// POST, not GET, deliberately: it is the only way the ServeHTTP-level CSRF check applies, and a +// secret must not be retrievable by URL alone (prefetch, history, referrer). +// +// SECRET DISCIPLINE: the plaintext goes into the JSON response body and nowhere else — never the +// hub log, never the event message or details_json. +func (s *Server) handleHostRevealRecoveryCredential(w http.ResponseWriter, r *http.Request, hostID string) { + host, err := s.store.GetHost(hostID) + if err != nil { + s.logger.Printf("[ERROR] reveal recovery credential %s: %v", hostID, err) + http.Error(w, "Internal error", http.StatusInternalServerError) + return + } + if host == nil { + http.NotFound(w, r) + return + } + cred, err := s.store.GetHostRecoveryCredential(hostID) + if err != nil { + s.logger.Printf("[ERROR] reveal recovery credential %s: %v", hostID, err) + http.Error(w, "Internal error", http.StatusInternalServerError) + return + } + if cred == nil { + // A 404 is not an access — nothing was delivered, so nothing is recorded on the timeline. + s.logger.Printf("[INFO] reveal recovery credential %s: no credential vaulted", hostID) + http.Error(w, "No recovery credential vaulted for this host", http.StatusNotFound) + return + } + // Transparency by default, exactly as handleRequestLogTail does it: SaveEvent alone writes the + // customer-visible timeline row WITHOUT emailing anyone (no dispatcher call here, by design). + // An unbound host has no customer to tell — the [INFO] line below is then the only record. + if host.CustomerID != "" { + if _, err := s.store.SaveEvent(host.CustomerID, "recovery_credential_revealed", "info", + "Az üzemeltető lekérte a géped konzolos hozzáférési jelszavát (távoli hibaelhárítás).", "", "hub"); err != nil { + s.logger.Printf("[WARN] SaveEvent recovery_credential_revealed %s/%s: %v", host.CustomerID, hostID, err) + } + } + s.logger.Printf("[INFO] operator revealed break-glass console credential for host %s (user=%s, secret %d chars)", + hostID, cred.Username, len(cred.Secret)) + w.Header().Set("Cache-Control", "no-store") + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "host_id": cred.HostID, + "username": cred.Username, + "password": cred.Secret, + "set_at": cred.SetAt.UTC().Format(time.RFC3339), + }) +} + // handleHostDelete — POST /hosts/{id}/delete (v0.47.0 stale host removal). Gates, in order: // - unknown host → 404 // - ONLINE host → 409 unconditionally (host reports authenticate via GetHostByAPIKey; diff --git a/hub/internal/web/hosts_recovery_reveal_test.go b/hub/internal/web/hosts_recovery_reveal_test.go new file mode 100644 index 0000000..3cf6d11 --- /dev/null +++ b/hub/internal/web/hosts_recovery_reveal_test.go @@ -0,0 +1,360 @@ +package web + +// Scenarios A–G (hub v0.84.0) — the break-glass Console access card + the operator-session reveal +// endpoint. +// +// SEAM DISCIPLINE: every test here drives RequireAuth(ServeHTTP), never a handler function +// directly. Scenario E (route ordering) is INVISIBLE to a handler-level test — the handler is +// correct and simply never runs — and that is the shape of the inert-seam defects on record. +// +// The load-bearing assertion is negative in the way that matters: the plaintext must not appear +// ANYWHERE in a rendered host page. revealCanary is deliberately distinctive so an accidental leak +// is greppable across the tree. + +import ( + "bytes" + "encoding/json" + "io" + "log" + "net/http" + "net/http/httptest" + "path/filepath" + "strings" + "testing" + "time" + + "golang.org/x/crypto/bcrypt" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +const revealCanary = "REVEAL-CANARY-9f2b7c" + +// newRevealServer builds a hub with a live operator password (so RequireAuth + the ServeHTTP CSRF +// check are both armed) and a CAPTURED logger, so the "the secret never reaches the log" assertion +// has something to read. +func newRevealServer(t *testing.T) (*Server, *store.Store, *bytes.Buffer) { + t.Helper() + var logBuf bytes.Buffer + st, err := store.New(filepath.Join(t.TempDir(), "t.db"), log.New(io.Discard, "", 0)) + if err != nil { + t.Fatalf("store.New: %v", err) + } + t.Cleanup(func() { st.Close() }) + s := New(st, "", "", "test", 30*time.Minute, log.New(&logBuf, "", 0)) + h, err := bcrypt.GenerateFromPassword([]byte("operator-pw"), bcrypt.MinCost) + if err != nil { + t.Fatal(err) + } + s.configPasswordHash = string(h) + return s, st, &logBuf +} + +// newRevealSession mints a live operator session and returns (cookie, csrfToken). +func newRevealSession(t *testing.T, s *Server) (*http.Cookie, string) { + t.Helper() + s.sessionsMu.Lock() + s.sessions["sess-token-reveal"] = &hubSession{ + expiresAt: time.Now().Add(time.Hour), + csrfToken: "csrf-token-reveal", + } + s.sessionsMu.Unlock() + return &http.Cookie{Name: "hub_session", Value: "sess-token-reveal"}, "csrf-token-reveal" +} + +// seedRevealHost creates a host (optionally bound to a customer) with a vaulted credential. +func seedRevealHost(t *testing.T, st *store.Store, hostID, customerID, secret string) { + t.Helper() + if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "k-" + hostID}); err != nil { + t.Fatal(err) + } + if secret != "" { + if err := st.SaveHostRecoveryCredential(hostID, "root@pam", secret); err != nil { + t.Fatal(err) + } + } +} + +// serveReveal drives the REAL stack: RequireAuth → ServeHTTP → routing → handler. +func serveReveal(t *testing.T, s *Server, req *http.Request) *httptest.ResponseRecorder { + t.Helper() + rr := httptest.NewRecorder() + s.RequireAuth(http.HandlerFunc(s.ServeHTTP)).ServeHTTP(rr, req) + return rr +} + +func countEvents(t *testing.T, st *store.Store, customerID, eventType string) int { + t.Helper() + evs, err := st.GetRecentEvents(customerID, 100) + if err != nil { + t.Fatal(err) + } + n := 0 + for _, e := range evs { + if e.EventType == eventType { + n++ + } + } + return n +} + +// --- Scenario A: the rendered host page carries presence + username + set_at, NEVER the secret --- +// RED-PROOF A: add `data-secret="{{.RecoverySecret}}"` to the card and a "RecoverySecret" key +// holding cred.Secret to hostDetailData → the canary assertion below goes RED. +func TestReveal_A_PageNeverCarriesTheSecret(t *testing.T) { + s, st, _ := newRevealServer(t) + cookie, _ := newRevealSession(t, s) + seedRevealHost(t, st, "demo-felhom-8363b5", "demo-felhom", revealCanary) + + req := httptest.NewRequest(http.MethodGet, "/hosts/demo-felhom-8363b5", nil) + req.AddCookie(cookie) + rr := serveReveal(t, s, req) + if rr.Code != http.StatusOK { + t.Fatalf("host page = %d, want 200", rr.Code) + } + body := rr.Body.String() + + // THE load-bearing assertion — the whole response, attributes, comments, scripts and all. + if strings.Contains(body, revealCanary) { + t.Fatal("SECRET LEAK: the vaulted console password appears in the rendered host page") + } + for _, want := range []string{ + "Console access", + "root@pam", + "/hosts/demo-felhom-8363b5/reveal-recovery-credential", + "Reveal", + } { + if !strings.Contains(body, want) { + t.Errorf("host page is missing %q", want) + } + } +} + +// --- Scenario B: reveal delivers the secret, records exactly one event, and never logs it --- +// RED-PROOF B: delete the SaveEvent call in handleHostRevealRecoveryCredential → the event +// assertion goes RED. +func TestReveal_B_RevealDeliversAndAudits(t *testing.T) { + s, st, logBuf := newRevealServer(t) + cookie, csrf := newRevealSession(t, s) + seedRevealHost(t, st, "demo-felhom-8363b5", "demo-felhom", revealCanary) + + req := httptest.NewRequest(http.MethodPost, "/hosts/demo-felhom-8363b5/reveal-recovery-credential", nil) + req.AddCookie(cookie) + req.Header.Set("X-CSRF-Token", csrf) + rr := serveReveal(t, s, req) + + if rr.Code != http.StatusOK { + t.Fatalf("reveal = %d, want 200 (body %q)", rr.Code, rr.Body.String()) + } + if got := rr.Header().Get("Cache-Control"); got != "no-store" { + t.Errorf("Cache-Control = %q, want no-store", got) + } + if ct := rr.Header().Get("Content-Type"); !strings.HasPrefix(ct, "application/json") { + t.Errorf("Content-Type = %q, want application/json", ct) + } + var got struct { + HostID string `json:"host_id"` + Username string `json:"username"` + Password string `json:"password"` + SetAt string `json:"set_at"` + } + if err := json.Unmarshal(rr.Body.Bytes(), &got); err != nil { + t.Fatalf("decode: %v (body %q)", err, rr.Body.String()) + } + if got.Password != revealCanary { + t.Errorf("password = %q, want the vaulted secret", got.Password) + } + if got.Username != "root@pam" || got.HostID != "demo-felhom-8363b5" { + t.Errorf("unexpected payload: %+v", got) + } + if got.SetAt == "" { + t.Error("set_at missing — the operator cannot judge staleness without it") + } + + // Exactly ONE timeline row, on the bound customer, info severity, source "hub", Hungarian. + evs, err := st.GetRecentEvents("demo-felhom", 100) + if err != nil { + t.Fatal(err) + } + n := 0 + for _, e := range evs { + if e.EventType != "recovery_credential_revealed" { + continue + } + n++ + if e.Severity != "info" { + t.Errorf("severity = %q, want info", e.Severity) + } + if e.Source != "hub" { + t.Errorf("source = %q, want hub", e.Source) + } + if !strings.Contains(e.Message, "konzolos hozzáférési jelszavát") { + t.Errorf("message is not the Hungarian customer-facing line: %q", e.Message) + } + if strings.Contains(e.Message, revealCanary) || strings.Contains(e.DetailsJSON, revealCanary) { + t.Fatal("SECRET LEAK: the password is in the event row") + } + } + if n != 1 { + t.Errorf("recovery_credential_revealed rows = %d, want exactly 1", n) + } + + // The hub log records the access — username + length only. + if strings.Contains(logBuf.String(), revealCanary) { + t.Fatal("SECRET LEAK: the password reached the hub log") + } + if !strings.Contains(logBuf.String(), "operator revealed break-glass console credential") { + t.Errorf("the access is not recorded in the hub log: %q", logBuf.String()) + } + + // No dispatcher call exists on this path — nothing was emailed. + notifs, err := st.GetRecentNotifications("demo-felhom", 100) + if err != nil { + t.Fatal(err) + } + if len(notifs) != 0 { + t.Errorf("reveal produced %d notification_log row(s); it must email nobody", len(notifs)) + } +} + +// --- Scenario C: nothing vaulted → the explanatory card, no control, 404 on POST, zero events --- +func TestReveal_C_NotVaulted(t *testing.T) { + s, st, _ := newRevealServer(t) + cookie, csrf := newRevealSession(t, s) + seedRevealHost(t, st, "sess-f-2670b5", "sess-f", "") // no credential row + + req := httptest.NewRequest(http.MethodGet, "/hosts/sess-f-2670b5", nil) + req.AddCookie(cookie) + body := serveReveal(t, s, req).Body.String() + if !strings.Contains(body, "not vaulted") { + t.Error("the not-vaulted state does not render its badge") + } + if !strings.Contains(body, "byo host") { + t.Error("the not-vaulted state does not explain WHY (byo / step 4b)") + } + if strings.Contains(body, "/hosts/sess-f-2670b5/reveal-recovery-credential") { + t.Error("a Reveal control is offered for a host with nothing vaulted") + } + + req = httptest.NewRequest(http.MethodPost, "/hosts/sess-f-2670b5/reveal-recovery-credential", nil) + req.AddCookie(cookie) + req.Header.Set("X-CSRF-Token", csrf) + rr := serveReveal(t, s, req) + if rr.Code != http.StatusNotFound { + t.Fatalf("reveal on an unvaulted host = %d, want 404", rr.Code) + } + // A 404 is not an access: only a DELIVERED secret is recorded. + if n := countEvents(t, st, "sess-f", "recovery_credential_revealed"); n != 0 { + t.Errorf("a 404 wrote %d event row(s); it must write none", n) + } +} + +// --- Scenario D: the CSRF gate (security) --- +// RED-PROOF D: the companion below sends the token and asserts 200, proving this test +// discriminates on CSRF rather than passing because the request was malformed some other way. +func TestReveal_D_CSRFRequired(t *testing.T) { + s, st, _ := newRevealServer(t) + cookie, csrf := newRevealSession(t, s) + seedRevealHost(t, st, "demo-felhom-8363b5", "demo-felhom", revealCanary) + + req := httptest.NewRequest(http.MethodPost, "/hosts/demo-felhom-8363b5/reveal-recovery-credential", nil) + req.AddCookie(cookie) // session, but NO CSRF token + rr := serveReveal(t, s, req) + if rr.Code != http.StatusForbidden { + t.Fatalf("reveal without CSRF = %d, want 403", rr.Code) + } + if strings.Contains(rr.Body.String(), revealCanary) { + t.Fatal("SECRET LEAK: the 403 body carries the password") + } + if n := countEvents(t, st, "demo-felhom", "recovery_credential_revealed"); n != 0 { + t.Errorf("a CSRF refusal wrote %d event row(s); it must write none", n) + } + + // The discriminator: the SAME request WITH the token succeeds. + req = httptest.NewRequest(http.MethodPost, "/hosts/demo-felhom-8363b5/reveal-recovery-credential", nil) + req.AddCookie(cookie) + req.Header.Set("X-CSRF-Token", csrf) + if rr := serveReveal(t, s, req); rr.Code != http.StatusOK { + t.Fatalf("the same request WITH a CSRF token = %d, want 200 — the 403 above proves nothing", rr.Code) + } +} + +// --- Scenario E: the method gate — and, through it, the ROUTE ORDER (seam test) --- +// RED-PROOF E: move the new case BELOW `case strings.HasPrefix(path, "/hosts/")` → the GET falls +// through to the catch-all, renders the host detail page 200, and this test goes RED. +func TestReveal_E_MethodGateAndRouteOrder(t *testing.T) { + s, st, _ := newRevealServer(t) + cookie, _ := newRevealSession(t, s) + seedRevealHost(t, st, "demo-felhom-8363b5", "demo-felhom", revealCanary) + + req := httptest.NewRequest(http.MethodGet, "/hosts/demo-felhom-8363b5/reveal-recovery-credential", nil) + req.AddCookie(cookie) + rr := serveReveal(t, s, req) + if rr.Code != http.StatusMethodNotAllowed { + t.Fatalf("GET on the reveal route = %d, want 405", rr.Code) + } + // The route-order half: a fall-through to /hosts/ would render the host page instead. + if strings.Contains(rr.Body.String(), "Console access") { + t.Fatal("the reveal route fell through to the /hosts/ catch-all — the case is registered AFTER it") + } + if strings.Contains(rr.Body.String(), revealCanary) { + t.Fatal("SECRET LEAK: the 405 body carries the password") + } +} + +// --- Scenario F: unknown host --- +func TestReveal_F_UnknownHost(t *testing.T) { + s, _, _ := newRevealServer(t) + cookie, csrf := newRevealSession(t, s) + + req := httptest.NewRequest(http.MethodPost, "/hosts/does-not-exist/reveal-recovery-credential", nil) + req.AddCookie(cookie) + req.Header.Set("X-CSRF-Token", csrf) + if rr := serveReveal(t, s, req); rr.Code != http.StatusNotFound { + t.Fatalf("reveal on an unknown host = %d, want 404", rr.Code) + } +} + +// --- Scenario G: unauthenticated --- +func TestReveal_G_Unauthenticated(t *testing.T) { + s, st, _ := newRevealServer(t) + seedRevealHost(t, st, "demo-felhom-8363b5", "demo-felhom", revealCanary) + + req := httptest.NewRequest(http.MethodPost, "/hosts/demo-felhom-8363b5/reveal-recovery-credential", nil) + req.Header.Set("X-Requested-With", "XMLHttpRequest") // API-like → 401 rather than a login redirect + rr := serveReveal(t, s, req) + if rr.Code != http.StatusUnauthorized { + t.Fatalf("unauthenticated reveal = %d, want 401", rr.Code) + } + if strings.Contains(rr.Body.String(), revealCanary) { + t.Fatal("SECRET LEAK: the 401 body carries the password") + } + if n := countEvents(t, st, "demo-felhom", "recovery_credential_revealed"); n != 0 { + t.Errorf("an unauthenticated call wrote %d event row(s); it must write none", n) + } +} + +// --- Edge case (§8): an UNBOUND host reveals fine and writes no event (SaveEvent needs a customer) --- +func TestReveal_UnboundHostRevealsWithoutAnEvent(t *testing.T) { + s, st, logBuf := newRevealServer(t) + cookie, csrf := newRevealSession(t, s) + seedRevealHost(t, st, "unbound-01", "", revealCanary) + + req := httptest.NewRequest(http.MethodPost, "/hosts/unbound-01/reveal-recovery-credential", nil) + req.AddCookie(cookie) + req.Header.Set("X-CSRF-Token", csrf) + rr := serveReveal(t, s, req) + if rr.Code != http.StatusOK { + t.Fatalf("reveal on an unbound host = %d, want 200", rr.Code) + } + if !strings.Contains(rr.Body.String(), revealCanary) { + t.Error("the unbound host's secret was not delivered") + } + // No placeholder customer id is invented — the hub log is the only record. + if n := countEvents(t, st, "", "recovery_credential_revealed"); n != 0 { + t.Errorf("an unbound host wrote %d event row(s) against an empty customer id", n) + } + if !strings.Contains(logBuf.String(), "unbound-01") { + t.Error("the unbound host's reveal is recorded nowhere at all") + } +} diff --git a/hub/internal/web/server.go b/hub/internal/web/server.go index cbf7033..f1736c7 100644 --- a/hub/internal/web/server.go +++ b/hub/internal/web/server.go @@ -364,6 +364,15 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { } else { http.Error(w, "Method not allowed", http.StatusMethodNotAllowed) } + // v0.84.0 break-glass console credential — suffix route BEFORE the bare /hosts/ catch-all + // (registered after it, the POST would 404 and the GET would silently render the host page). + case strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/reveal-recovery-credential"): + hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/reveal-recovery-credential") + if r.Method == http.MethodPost { + s.handleHostRevealRecoveryCredential(w, r, hostID) + } else { + http.Error(w, "Method not allowed", http.StatusMethodNotAllowed) + } case strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/request-logs"): hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/request-logs") if r.Method == http.MethodPost { diff --git a/hub/internal/web/templates/host_detail_body.html b/hub/internal/web/templates/host_detail_body.html index cd61142..15bb3a6 100644 --- a/hub/internal/web/templates/host_detail_body.html +++ b/hub/internal/web/templates/host_detail_body.html @@ -279,6 +279,96 @@ + +
+

Console access

+ {{if .RecoveryVaulted}} +
+
+ User + {{.RecoveryUsername}} +
+
+ Password set + {{timeAgo .RecoverySetAt}} +
+
+
+ •••••••••••••••• + + +
+

+ Break-glass credential for the PVE web console at https://<host-ip>:8006 (realm: Linux PAM standard authentication). Revealing it is recorded on the customer's event timeline. Last vaulted value — if root@pam was changed on the box without re-vaulting, this is stale. +

+ {{else}} +

not vaulted

+

+ No console credential is vaulted for this host. Expected for a byo host — the owner manages root@pam. Otherwise the installer's step 4b did not run; re-run felhom-host-install.sh, or set and vault one per the break-glass runbook §5. +

+ {{end}} +
+ {{if .RecoveryVaulted}} + + {{end}} + {{if .Deletable}}