admin
a91d0b2421
gates / gates (push) Successful in 13m1s
Spike: per-app normal levels on three boxes (19-31 days), the off-site push as a line measure (a lower bound), qBittorrent/Transmission/Plex explainers measured on 9202, a tc ceiling measured (not built). Live on 9202 with 0.342.0-rc.1: limit set; real seed explained; CPU burner and the hidden flood alarmed once each and cleared once; the household's limit left alone. Teardown: apps removed, 9202 back on 0.341.0, receiver 9471 destroyed.
114 lines
11 KiB
Markdown
114 lines
11 KiB
Markdown
# The hub's last plain tokens, sealed (R-975, R-1241's Felhom half) — 2026-10-23 night
|
||
|
||
Class: implementation (hub 0.168.0, built, **not deployed**) + a design for the backup copies (Part 1.4).
|
||
Architecture read: `architecture/05-hub-architecture.md` §16.2–§16.3 (sealing, the snapshot), `13-threat-model.md`
|
||
(attacker j, R-1241), `audits/plex2shell-felhom-impact-2026-10-23/README.md`. Names and places only — no value appears
|
||
here or in any file of this session. Every count below came from a read-only copy of the hub's own 14:00Z snapshot
|
||
(`VACUUM INTO`, so it holds every committed write), copied to a 0700 scratch folder and deleted after.
|
||
|
||
## 1. Every hub column that holds a secret (measured, 2026-10-23)
|
||
|
||
| Where | What | Before 0.168.0 | 0.168.0 |
|
||
|---|---|---|---|
|
||
| `hosts.api_key` | each box's hub key | sealed (R-879, 0.138.0) — 4/4 | same |
|
||
| `customer_configs.api_key`, `retrieval_password` | controller key, owner passphrase | sealed (R-879) — 5/5 each | same |
|
||
| `host_pbs_secrets.value` | PBS-DR token | sealed (R-879) — 4/4 | same |
|
||
| `one_time_secrets.value` | Storage Box sub-account password | sealed (R-821, 0.127.0) — 4/4 | same |
|
||
| `host_recovery.secret` | break-glass console password | sealed (R-133, 0.135.0) — 4/4 | same |
|
||
| **`customer_configs.config_json` → `infrastructure.cf_tunnel_token`** | Cloudflare tunnel token | **plain — 4 rows** | **sealed** |
|
||
| **`customer_configs.config_json` → `infrastructure.cf_api_token`** | Cloudflare API token | **plain — 4 rows** | **sealed** |
|
||
| **`customer_configs.config_json` → `git.token`** | git token (the form still accepts one) | none stored today | **sealed** when written |
|
||
| `host_escrow.blob`, `identity_blob`; `host_escrow_superseded.*` | escrow | age-wrapped by the household's recovery code — not the hub's to seal | same |
|
||
| `api_tokens.token_hash`, `selfbind_tokens.token_hash`, `appliance_registrations.token_hash`, `customer_claims.code_hash`, `hub_settings[operator_password_hash]` | operator tokens, bind tokens, claim codes, operator password | one-way hashes (SHA-256 / bcrypt) | same |
|
||
| `hosts.api_key_hash`, `customer_configs.api_key_hash` | lookup twins | unkeyed SHA-256 of a 256-bit random key (R-879) | same |
|
||
| `wg_peers.pubkey`, `wg_endpoints.server_pubkey`, `host_escrow.key_fingerprint`, `appliance_registrations.ssh_host_pubkeys`, `hub_settings[oob_operator_ssh_pubkey]` | public keys, fingerprints | not secret | same |
|
||
| `guests.api_key` | unused column | empty 7/7 | same |
|
||
| `appliance_registrations.pairing_code` | 6-character console code | plain, 31 rows | **NOT-A-FINDING:** it is printed on the box's screen and is only one of two factors; the other (the self-bind token) is stored as a hash. Pending registrations only. |
|
||
|
||
**Do the token VALUES sit anywhere else?** Each of the 8 distinct Cloudflare values was searched (by value, in memory,
|
||
never printed) in every column of every other table: **0 hits**. The key NAME `cf_tunnel_token` appears in one
|
||
`reports.report_json` and one `app_telemetry.issues_json` row — names, not values. `events.details_json` carries
|
||
`repo_key` (literally the word „changed") and PBS `token_id`s (a name like `user@pbs!name`, not a secret).
|
||
|
||
## 2. The change (hub 0.168.0)
|
||
|
||
- **Same mechanism, same key** (`05` §16.2): each token leaf in `config_json` holds `enc:v1:` + AES-256-GCM under
|
||
`OFFSITE_SECRET_KEY`. The rest of the JSON stays readable. `internal/store/r975_config_token_seal.go`.
|
||
- **New writes sealed:** `SaveCustomerConfig` (the only writer of `config_json`). No key and a token → refused.
|
||
- **Every reader opens through the existing helper:** `openCustomerSecrets`, which the three readers
|
||
(`GetCustomerConfig`, `GetCustomerConfigByAPIKey`, `ListCustomerConfigs`) already call. A leaf that does not open is
|
||
blanked and sets `SecretsUnreadable` (R-879's rule: never served, a save refuses the record, `configgen.Generate` refuses).
|
||
- **Old rows sealed once at start-up:** `SealLegacyConfigTokens`, logged as `config tokens sealed at rest (N legacy
|
||
plaintext token(s) sealed now)` — counts only. **No `config_version` bump** — the box's config is byte-for-byte the
|
||
same, so no box re-pulls or restarts. Expected on the live hub: N = 8 on the first start, 0 after.
|
||
- **Then one `VACUUM` + WAL checkpoint** (only when N > 0): an `UPDATE` leaves the old plaintext row in a freed block of
|
||
`hub.db` (the SQLite driver's `secure_delete` is 0 — measured). Timed on a copy of the live 534 MB database: **2.8 s**,
|
||
well inside the liveness probe's ~95 s. The volume has 2.4 GB free for it.
|
||
- **Roll-back:** `felhom-hub -unseal-box-secrets` now also opens these leaves (all or nothing), so going back to
|
||
0.167.0 stays the documented procedure (`05` §16.2). Without it an older hub would hand the box the sealed string as its
|
||
tunnel token.
|
||
|
||
**Tests, each red-proved** (the rule broken on purpose → the test failed → restored):
|
||
|
||
| Test | Pins | Red-proof (mutation → failed) |
|
||
|---|---|---|
|
||
| `TestR975_FreshWriteIsStoredSealed` | a fresh write is sealed; non-secret fields intact; no key → refused | write path unsealed → FAIL (with 5 others) |
|
||
| `TestR975_ReaderGetsTheSameValue` | all three readers + a load-modify-save return the same value | open on read removed → FAIL |
|
||
| `TestR975_MigrationSealsOnceThenNoOp` | 3 sealed, then 0; no `config_version`/`updated_at` change | migration no-op → FAIL; migration bumps the version → FAIL |
|
||
| `TestR975_DBFileWithoutKeyRevealsNoToken` | **a copy of `hub.db` + `-wal` + `-shm` greps clean**; positive control: the pre-0.168.0 rows DO show in the file | VACUUM removed → FAIL (the old row stays in a freed block) |
|
||
| `TestR975_WrongKeyBlanksAndRefusesSave` | wrong key → blanked, flagged, save refused, right key restores | not blanked → FAIL; flag ignores the tokens → FAIL |
|
||
| `TestR975_UnsealRestoresPlaintextForAnOlderHub` | roll-back opens the tokens, all or nothing | the config half skipped → FAIL |
|
||
| `TestR975_EveryFormSecretIsSealed` (web) | every leaf the operator form writes is either on a 2-name non-secret list or sealed | write path unsealed → FAIL |
|
||
| `TestR975_GeneratedConfigCarriesThePlainTunnelToken` (configgen) | the box still gets the REAL token in `controller.yaml` | open on read removed → FAIL |
|
||
|
||
Two first mutations broke the build instead of a test (an unused variable) and were redone so a test, not the
|
||
compiler, failed. The first version of the file test passed with the VACUUM removed — the token row was the last cell on
|
||
its page, so the new row overwrote it; a second customer written after it (as on the live hub) made the test honest.
|
||
|
||
**Run on a private copy of the real database** (the 14:00Z snapshot, in a 0700 scratch folder, with a THROWAWAY key —
|
||
the real one was never read; a temporary test file, deleted, never committed): the start-up sealing sealed **8** tokens
|
||
in **3.15 s** (incl. the VACUUM), a second run sealed **0**, all **5 of 5** customers' `config_json` read back equal to
|
||
the original (JSON-equal), and **0 of the 8** token values were found anywhere in the database files afterwards. A first
|
||
attempt compared whole records and showed „0 of 5 equal": the R-879 columns were sealed with the REAL key, so with the
|
||
throwaway key every record was flagged unreadable — the expected behaviour, not a defect; the second run compared the
|
||
JSON only. The copy was shredded after. Image: `gitea.dooplex.hu/admin/felhom-hub:0.168.0`
|
||
(`sha256:3bfdecaae527d4762059600cdfaad83daf4e4b5b20661fc47acf128180d87cb3`), from `5277d859`, CI run 2269 success.
|
||
|
||
**Measured on the live file (count of the key name only):** `cf_tunnel_token` occurs 12 times both in the live
|
||
`hub.db` and in the clean `VACUUM INTO` snapshot, so the live file holds no extra old copies today.
|
||
|
||
## 3. The backup copies — design only (Part 1.4)
|
||
|
||
**What exists today (measured):**
|
||
|
||
| Copy | Where | Encrypted? | Holds the tokens? |
|
||
|---|---|---|---|
|
||
| Hub snapshots `<data>/snapshots/` (2, 6-hourly) | the hub's Longhorn volume | no (the volume is `encrypted=false`) | after 0.168.0: sealed |
|
||
| **Longhorn backups of the hub volume** | `/mnt/5_hdd/backup/longhorn-pvc` (the NFS folder of R-1241) — **2 backups: 2026-10-18 03:00Z (weekly) and 2026-10-23 02:01Z (daily)**, ~4.1 GB each, `retain 1` each | **no** | **yes, in plaintext** — both predate 0.168.0 |
|
||
| The hub's own copy to ep0 (`felhom-hub-db-backup`, 6-hourly :30) | ep0 PBS, namespace `operator` | **yes, at rest:** 16 snapshots 2026-10-10 → 10-23, every `hubdb.pxar.didx` and catalog `crypt-mode: encrypt` (`index.json.blob` sign-only). The key `/etc/felhom-hub-backup/enc.key` is on DooPlex's **root disk (sdb1)**, not on the NFS disk, and on paper (S2). **In transit:** SSH port-forward `127.0.0.1:18007` + PBS TLS with a pinned fingerprint. Last push 14:31 CEST today, 7 s; the weekly restore-test last passed 2026-10-18 („4 sealed console password(s), 0 readable"). | sealed after 0.168.0 |
|
||
|
||
**The two options:**
|
||
|
||
| | (a) Encrypt the hub's Longhorn volume | (b) Take the hub volume out of the Longhorn NFS backup |
|
||
|---|---|---|
|
||
| What | A new Longhorn volume with `encrypted=true` (a StorageClass with a crypto Secret), copy `/data` across, point `manifests/hub.yaml` at it | Remove the volume from the `default` recurring-job group (one label on the Longhorn volume); the ep0 copy is the hub's only backup |
|
||
| Hub downtime | ~5–10 min (scale to 0, copy 1.5 GB, scale up) | none |
|
||
| Who | homelab (a StorageClass + Secret in `longhorn-system`) + a Felhom manifest change | homelab (the Longhorn volume label); no Felhom code |
|
||
| New key to keep | yes — the volume passphrase, on paper, and it lives in the cluster where the R-1241 attacker reaches the Longhorn API | none |
|
||
| What it fixes | the backups stay but are ciphertext; also the volume's freed blocks | no more hub copies in the NFS folder at all |
|
||
| What it loses | nothing | the hub's 90-day access log (`/data/access-log`, R-1140) and its local snapshots are no longer backed up anywhere; a lost volume is restored from ep0 (`RUNBOOK-hub-db-offsite-backup.md` §restore, restore-tested weekly) — at most ~6.5 h of hub data lost, the same as today's off-site promise |
|
||
| Does the encryption key sit next to the data? | yes (cluster Secret) | the PBS key is on another disk and on paper |
|
||
|
||
**CC's pick: (b)**, because the ep0 copy is already encrypted at rest and in transit (measured above), off-site, every 6
|
||
hours, and restore-tested weekly — the Longhorn copy adds a second, unencrypted copy in exactly the folder R-1241 found
|
||
open, and (a) would put its key in the same cluster. The access log's loss can be closed later by adding
|
||
`/data/access-log` to the ep0 push (small; not tonight). → decide row **R-1243**.
|
||
|
||
**The plain copies already in the folder** (the two hub-volume backups above): they hold today's tokens in plaintext.
|
||
Deleting backups is the operator's decision. Without action they rotate out by themselves after 0.168.0 is deployed —
|
||
the daily one at the next 04:00 run, the weekly one at the next Sunday 05:00 run (`retain 1` each) — **but only if the
|
||
volume stays in the backup job** (option a, or no decision). With option (b) the last two copies stay until someone
|
||
deletes them. Either way the tokens in them should then be considered exposed only if R-1241's door is used — the
|
||
Plex2Shell audit measured it was not. **Note:** Longhorn backs up the volume's blocks, so a block freed by the VACUUM can
|
||
still hold old plaintext in a later Longhorn backup; only (a) or (b) removes that — not measured, reasoned.
|