admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

hub v0.84.0 — break-glass console credential on the host page

Browse Source 
The credential existed and was not reachable when it was wanted. Every box has
had a strong random root@pam password since TASK G1, vaulted in the hub at day 0
and used for real during the sshd incident — but the only way to read it back was
a hand-written curl carrying the global operator key, a secret kept out-of-band.
In practice the PVE web console on a demo box felt locked.

The host page grows a Console access card: presence + username + set_at by
default, Reveal fetches the plaintext on demand for 60 s with a Copy button.
Masking clears the JS variable, and also fires on a second click and on
visibilitychange. A host with nothing vaulted says so, and says why.

The secret is NEVER rendered into the page, and that constraint shapes the
change. The render path uses a new store.GetHostRecoveryMeta whose struct and
SELECT both omit the secret column, so it is structurally incapable of carrying
one. The plaintext crosses the wire only in the response to POST
/hosts/{id}/reveal-recovery-credential (Cache-Control: no-store, CSRF-gated at
the ServeHTTP level; POST precisely so that gate applies and so no secret is
retrievable by URL alone). Deliberately NOT the customer page's data-secret
widget, which embeds the plaintext on every load.

A delivered reveal writes one recovery_credential_revealed event on the host's
customer timeline (info, source hub, Hungarian) via SaveEvent alone — no
dispatcher, nobody emailed, the log_tail_requested shape. Two reveals write two
events: the register records accesses, not states. A 404 is not an access. An
unbound host reveals fine and writes no event; the [INFO] hub line, carrying the
username and a length only, is then the record.

The global-key API path is untouched by design — it is the route for when the
hub UI itself is broken, and coupling it to the session layer would delete the
independence that makes it a fallback.

Recorded as a real trade: the hub session password alone now unlocks console root
fleet-wide, where retrieval previously also needed the global key. Accepted for a
single-operator, HU-geo-fenced hub that already stores these passwords in
plaintext at rest (CONTEXT.md ruling S-4). The plaintext-at-rest half is filed as
R-133 — every hub DB backup is a fleet-wide console-credential dump.

Tests 550 -> 559; four red-proofs (page leak, audit event, CSRF gate, route
order) each run, observed failing, and reverted. The route-order proof is a seam
test driving ServeHTTP: a handler-level test cannot see that defect, because the
handler is correct and simply never runs.
This commit is contained in:
authentik Default Admin
2026-07-31 08:19:36 +02:00
parent 0a9bd3829d
commit 1956e5d390
14 changed files with 801 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+101 -155
View File
@@ -1,183 +1,129 @@
# REPORT — E-2 increment 1: the backup target is now defended and audible (2026-07-29)
# REPORT — Hub v0.84.0: break-glass console credential on the host page (2026-07-31)
**Overwritten** per the standing rule. **E-2 is PARTIALLY shipped** — this increment is Parts 1 + 5
plus findings E-2b/E-2c. Parts 2, 3, 4 and E-2a remain open (see §6).
## 1. Confirmed baselines actually used
**Live fleet after this run:** hub **0.81.0**, agent **0.112.0** (both boxes), controller **0.184.1**
(both boxes). `peti-felhom` deliberately untouched.
| Repo | `main` @ start | Version before | Version after |
|---|---|---|---|
| felhom.eu | `0a9bd3829d9a4465b032ea78813f55e935fbf88e` ("D5 SHIPPED…", 2026-07-30) | hub **v0.83.0** (`manifests/hub.yaml:128` → `felhom-hub:0.83.0`) | hub **v0.84.0** |
---
Clean-tree gate at start: `git status --porcelain` empty, `HEAD == origin/main`. **Note:** a second
Claude Code session was working in this shared clone concurrently — see §10.
## 1. Why this order
## 2. Files created / modified
The operator reordered the plan, correctly: **E-2b and E-2c protect a configuration that exists
right now; E-2a builds for boxes that do not.** Both demo boxes are configured, Peti is parked, and
no new installs are happening this week — so Part 3 has no user yet, while E-1's new configuration
had a drive that could be ejected without warning and no event when it went absent.
Created:
- `hub/internal/web/hosts_recovery_reveal_test.go`
## 2. E-2c — the backup drive could be ejected out from under the backup
Modified:
- `hub/internal/store/host_recovery.go` — `HostRecoveryMeta` + `GetHostRecoveryMeta`
- `hub/internal/store/host_recovery_test.go` — Group H
- `hub/internal/web/hosts.go` — `handleHostRevealRecoveryCredential`; `hostDetailData` +3 keys
- `hub/internal/web/server.go` — route, above the bare `/hosts/` catch-all
- `hub/internal/web/templates/host_detail_body.html` — Console access card + fetch-on-demand script
- `hub/CHANGELOG.md`, `REUSE.md`, `CONTEXT.md` (ruling **S-4**),
`documentation/runbooks/break-glass.md` (§3.1 + §5),
`documentation/architecture/00-capability-map.md` (L120),
`documentation/backlog/ROADMAP.md` + `documentation/backlog/OPEN-ITEMS.md` (**R-133**)
`RoleForStorage` types a `local-dir` on a non-system device as **user-data**, so the pre-existing
eject role gate PASSED the drive E-1 had just made the vzdump target. `POST /disks/eject` on
`/mnt/nvme-1tb` or `/mnt/hdd_1` would have **succeeded silently**, taking the only local whole-guest
backup with it while the box kept reporting a configured tier.
## 3. Commits pushed to `main`
Fixed in agent v0.111.0/0.112.0: `handleDiskEject` and `handleDiskDecommission` refuse `409` when the
mount backs a configured tier, naming the storage and the remedy — the op is *ordered*, not
forbidden.
*(filled in at push — see `git log --oneline`)*
**Deliberately NOT a role reclassification.** Making `RoleForStorage` return `RoleBackup` would
refuse every legitimate eject of the customer's own data drive, because on both boxes that drive
**is** the target. `TestEjectStillAllowedOnANonTargetDrive` pins that, and the over-correction is
red-proofed.
## 4. Test results + red-proofs
**Live on both boxes:**
Every test drives `RequireAuth(ServeHTTP)` — never a handler function directly.
```
demo-hp felhom-backup mount=/mnt/nvme-1tb role=user-data backup_target=True
demo-felhom felhom-backup mount=/mnt/hdd_1 role=user-data backup_target=True
```
| Test | Scenario | Result |
|---|---|---|
| `TestReveal_A_PageNeverCarriesTheSecret` | A — render, canary absent from the WHOLE body | PASS |
| `TestReveal_B_RevealDeliversAndAudits` | B — 200 + no-store + payload + exactly 1 event + no log leak + 0 notifications | PASS |
| `TestReveal_C_NotVaulted` | C — explanatory card, no control, 404, zero events | PASS |
| `TestReveal_D_CSRFRequired` | D — 403, no leak, zero events; + the with-token discriminator | PASS |
| `TestReveal_E_MethodGateAndRouteOrder` | E — 405 and no fall-through to the host page | PASS |
| `TestReveal_F_UnknownHost` | F — 404, no panic | PASS |
| `TestReveal_G_Unauthenticated` | G — 401, no leak, zero events | PASS |
| `TestReveal_UnboundHostRevealsWithoutAnEvent` | §8 edge — 200, no event, log is the record | PASS |
| `TestGetHostRecoveryMeta_MetadataOnly` | H — store round-trip + absent cases | PASS |
That single line is the exposure: `role=user-data` (which the gate permits) yet `backup_target=True`.
Both ejects returned **409** with the full message; both decommissions **409**; both drives still
mounted with `dump/` intact — the refusal had no side effect. `/var/lib/vz` is still refused by the
**pre-existing** role gate (`role: system`), proving the new gate is narrow.
**Red-proofs — each mutation applied, observed failing, reverted:**
## 3. E-2b — the alarm that was defined and never called
| # | Mutation | Observed | Reverted |
|---|---|---|---|
| A | `hostDetailData` gains `RecoverySecret` (from `GetHostRecoveryCredential`) + `data-secret="{{.RecoverySecret}}"` on the card | `FAIL … SECRET LEAK: the vaulted console password appears in the rendered host page` | yes |
| B | delete the `SaveEvent` block in `handleHostRevealRecoveryCredential` | `FAIL … recovery_credential_revealed rows = 0, want exactly 1` | yes |
| D | short-circuit the `ServeHTTP` CSRF check (`if false && !s.validateCSRF(r)`) | `FAIL … reveal without CSRF = 200, want 403` | yes |
| E | move the new `case` BELOW `case strings.HasPrefix(path, "/hosts/")` | `FAIL … GET on the reveal route = 404, want 405` | yes |
`NotifyStorageDisconnected` and `NotifyStorageReconnected` were registered in `allowedEventTypes`, in
`DefaultEnabledEvents`, and given Hungarian hub copy — and **called from nowhere**. A drive going
absent produced stopped apps, a `[WARN]` log and a UI badge, then silence on every channel. A drive
that is *only* a backup target has no apps to stop, so it was silent twice over. Fifth instance of
this class; found by Phase 0, not by a failure.
**Red-proof A did not land on the first attempt, and that is worth recording.** The first mutation
edited only the template; a `{{.RecoverySecret}}` against a map with no such key renders empty, so
the test passed and would have certified nothing. The proof landed only once the *data* half was
mutated too — i.e. the assertion is pinned to the view-model, not to template text.
Now wired in `ReconcileDriveGates`, with the target case raising the more specific
`backup_target_absent` (error) and recovering as `backup_target_restored` (info — the pairing-gated
pattern; `severityNotifies` NOT widened). Never both: two mails for one event trains people to
ignore the channel.
**Red-proof E's symptom differed from the prediction.** The spec expected the misordered route to
render the host page (200); it actually 404s, because the catch-all takes
`demo-felhom-8363b5/reveal-recovery-credential` as the host id and `GetHost` misses. The test goes
red either way, and its second assertion (no `Console access` in the body) still pins the
fall-through case.
## 4. The bug I introduced and caught before deploy
**One spec assertion caught a real gap during development.** Scenario A's requirement that the page
"contains a Reveal control targeting `/hosts/{id}/reveal-recovery-credential`" failed at first: the
URL was assembled in JS (`'/hosts/' + encodeURIComponent(hostID) + '/…'`) and appeared nowhere in the
DOM. Fixed by putting the endpoint in `data-reveal-url` on the button **and making the fetch read it
from there**, so the string the render test asserts is the string the request uses — an attribute
nothing reads would have been a hollow assertion.
`ReconcileDriveGates` resolves the target as `isTarget[a.Path]`, and **`a.Path` is the registered
GUEST path** (`/mnt/felhom-drives/<name>`), not the agent's host `MountPath`. I had keyed the map on
`MountPath` alone, so the backup-target branch was **unreachable** — every absent drive, the target
included, would have fallen through to the generic event.
## 5. Test count
It would have looked wired, passed its own unit tests, and shipped silently wrong on exactly the
drive it exists for. Caught by tracing `a.Path` back to its source instead of assuming. Fixed in
v0.184.1 (both keys, mirroring `planDriveGates`), red-proofed, and the **0.184.0 image is superseded
and was never deployed**.
**550 → 559** (8 web + 1 store). Full suite `go build ./... && go vet ./... && go test ./...` in
`hub/`: **rc=0, all 17 packages ok**. Run as a separate command from the commit, per standing rule 1.
## 5. Tests and red-proofs
## 6. Deployed version
| Red-proof | Observed failure |
|---|---|
| hub allowlist entry removed | `backup_target_absent must be in allowedEventTypes, or POST /event 400s…` |
| hub Hungarian copy shortened | `must state the consequence…, got: "Meghajtó hiányzik."` |
| C — auto-elevation on register | `registering drives assigned the backup target "/mnt/hdd_1"` |
| E-2c — eject guard removed | `eject of the backup-target drive SUCCEEDED (200)` |
| E-2c over-correction | the gate blocked `/mnt/spare` (a non-target) |
| keying reverted to MountPath | `not resolvable by its GUEST path — …the branch would never fire` |
*(filled in after the build/manifest/sync steps)*
Every mutation was **verified to have landed** before the result was trusted. The first attempt
silently no-op'd (gofmt had realigned the map) and the test "passed" — a false proof.
## 7. NOT yet live-validated — awaiting the operator
Controller notify tests observe the **wire** (a real `Notifier` against an `httptest` hub), not a
mock, because the failure class is "nothing arrives".
**That the revealed password actually authenticates at `https://<box-ip>:8006` as `root@pam` on
`demo-felhom-8363b5`.** This is the only test that proves the hub's copy still matches the box, and
it needs a browser and a real login — CC has neither here (no `claude-in-chrome` on DooPlex). Every
other leg is endpoint-level validated (§6).
## 6. What is NOT done
## 8. Teardown
- **Part 2** (installer Case A/B), **Part 3** (the offer + the agent-side move), **Part 4** (the
degraded Hungarian banner). Phase 0 inverted their emphasis: the installer has **no drive-enrollment
step**, so at install the common case is system-drive-only and **Part 3 is the primary path**.
- **E-2a** — the guarded wrapper. The agent holds neither `Datastore.Allocate` at `/storage` nor
`Permissions.Modify`, and its sudoers has no `pvesm`/`pveum`. Use the wrapper pattern; **do not
widen the agent's PVE role.**
- **`backup_target_absent` is not proven END-TO-END live.** Its components are: the controller emits
the right type/severity (proven at the wire), the hub allowlists it (unit-pinned, image deployed),
and the keying is pinned. Proving the whole chain needs a live drive to actually go absent, which
means unmounting an enrolled drive carrying customer data and the guest bind on a remote box —
the same thing declined in E-1, for the same reason.
- Full drive-loss recovery (physical access) — carried forward from E-1.
**This run provisioned nothing** — no guest, no VM, no customer, no drive, no external resource. No
teardown obligation.
## 7. Observations
## 9. Backlog rows opened / closed / re-ranked
- **`peti-felhom` untouched.** Its whole-guest backup still shares a device with its guest, so a
drive failure there is offsite-only recovery. Accepted until the operator's reinstall; re-evaluate
if that slips past ~2026-09-01.
- The fleet was briefly split (demo-hp updated first). Left that way it would have protected one of
two identically-exposed boxes, so demo-felhom was rolled to the same proven artifacts.
- `StorageTarget.Role` carries a documented vocabulary including `vzdump-target` and is always `""`.
Checked — it is a deliberate unpopulated field ("hub-owned, slice 10"), **not** a sixth dead seam.
- **Opened: `R-133`** — the vaulted secret is plaintext at rest, so every hub DB backup is a
fleet-wide console-credential dump; envelope-encrypt `host_recovery.secret` under a KEK held
outside the DB. Added to `ROADMAP.md` and `OPEN-ITEMS.md` (owner CC, `READY (M)`). Named the
capability-map row it would flip.
- **Closed / re-ranked: none.**
- **ID collision, resolved:** the spec predicted R-128. The concurrent session's *uncommitted* WIP in
`OPEN-ITEMS.md` had already taken **R-128 through R-132**, so this item took **R-133**. An ID
register that lives in a file two sessions edit at once cannot allocate safely by reading committed
state — worth noting, not fixed here.
---
## 10. Observations — recorded, deliberately NOT acted on
# E-2 increment 2 (same day): the wrapper, the installer, the offer and the banner
**Live fleet:** hub **0.81.0**, agent **0.113.0**, controller **0.185.1**, host-install **1.22.0**
(script; no reinstall performed). Both demo boxes. `peti-felhom` untouched.
## E-2a — the guarded wrapper
`configs/felhom-backup-target-apply` behind a literal `FELHOM_BACKUPTARGET` sudoers alias. **The
agent's PVE role was not widened** — it still holds neither `Datastore.Allocate` at `/storage` nor
`Permissions.Modify`.
All five laws proven live as root on demo-hp, with **0 stray storages created**:
| Probe | Result |
|---|---|
| F-1 subdirectory (`/mnt/nvme-1tb/dump`) | REFUSED — "must be the drive's OWN mountpoint" |
| F-2 unmounted path | REFUSED — "would silently retarget onto the system drive" |
| root device (`/`) | REFUSED |
| idempotent re-apply on the real target | no-op, exit 0 |
| repointing an existing id | REFUSED |
`POST /backup/target` drives it **create → grant → config**, and deliberately does **not** restart the
agent: it returns `restart_required` so the restart happens behind the caller's own immediate
in-flight check. That is E-1's spurious-tier-failure mistake encoded as a contract, and a test asserts
the handler never restarts.
## Part 2 — the installer (v1.22.0)
Case A creates the target through the same wrapper; Case B records DEGRADED in two warning lines and
**still succeeds** — a single-drive appliance is valid. Scenario F returns early with **0 wrapper
calls**; the red-proof (guard removed) shows **2**, i.e. it would have "corrected" a correct box.
**Installer-logic-tested, NOT install-tested** — the harness runs the extracted functions with
stubbed `pvesm`/wrapper. No reinstall was performed.
## Parts 3+4 — the offer and the banner
`POST /api/storage/backup-target/assign` is the **only** writer of the role. `GET
/api/storage/backup-target` drives the banner. Hungarian copy is FACT → CONSEQUENCE → REMEDY.
**Scenario E proven live on both boxes** — healthy renders nothing:
```
demo-hp {"degraded":false,"known":true,"label":"NVME 1TB","target":"felhom-backup"}
demo-felhom {"degraded":false,"known":true,"label":"USB HDD 1TB","target":"felhom-backup"}
```
No `message` key at all. Red-proofed: adding a reassurance to the healthy branch fails Scenario E.
## Three defects I introduced and caught
1. **The offer endpoints were unreachable.** Registered as `/api/backup-target` inside
`ServeStorageAPI`, which `main.go` mounts only at `/api/storage/`. Every unit test passed because
they called the handlers directly; the **first live call** returned `endpoint not found`. Fixed in
v0.185.1 with a test that asserts the dispatcher's own source contains both routes.
2. **A hollow test.** `TestUnknownStateRendersNothing` used `{Known:false}` with `Degraded` left
false, so it passed even with the `!Known` guard deleted. Its own red-proof exposed it; the
fixture is now `{Known:false, Degraded:true}`.
3. **A gofmt-realignment no-op** (again): a `sed`-style replace silently matched nothing because the
formatter had changed the spacing. Every mutation is now verified to have landed before its result
is trusted.
## Still not live-proven
- The **DEGRADED banner** and the **offer acceptance** — both boxes are healthy, so neither state
occurs naturally. Unit- and red-proofed only.
- **`backup_target_absent` end-to-end** — needs a live enrolled drive to go absent.
- **Case A/B on a real install** — installer-logic-tested only.
- **Full drive-loss recovery** — physical access.
1. **`customer_unified.html`'s `data-secret` / `toggleSecret` widget embeds the plaintext in the page
HTML on every load.** It therefore lives in the back/forward cache, in "save page as", and in any
DOM-capturing screenshot. Acceptable for one customer's retrieval passphrase; it is the reason
this task built fetch-on-demand instead of reusing it. Not refactored — out of scope.
2. **`validateCSRF` returns `true` when no session cookie is present** (`server.go`, the Basic-Auth
path). So a Basic-Auth caller reaches the reveal endpoint **without any CSRF token**. That is the
pre-existing hub-wide contract, not something this endpoint introduced, and it is what makes the
§6 curl validation possible at all — but it does mean "CSRF-gated" is true only for session
callers. Recorded, not changed.
3. **The Console access card also renders on the customer page's Host tab**, because
`host_detail_body` is shared and `configs.go` builds its view models through the same
`hostDetailData`. That is per-host and operator-only (the hub has no customer login), so it does
not breach the "never on the hosts LIST" rule — but it is a second surface, and it is stated here
rather than left to be discovered.
4. **A concurrent session shares this clone.** Its in-flight edits to `OPEN-ITEMS.md`,
`RUNBOOK-manual-build.md`, `RUNBOOK-publish-0.79-0.110-2026-07-10.md` and two new untracked files
were **not** swept into this commit: every path was staged explicitly, and `OPEN-ITEMS.md` was
staged as a hand-built blob (`HEAD` + the R-133 row only) so the foreign WIP stayed unstaged in
the working tree. `REPORT.md` was taken here because that session had already chosen the
`REPORT-tester-gate-2026-07-31.md` sibling.