felhom.eu/documentation/audits/hub-token-seal-2026-10-23

The hub's last plain tokens, sealed (R-975, R-1241's Felhom half) — 2026-10-23 night

Class: implementation (hub 0.168.0, built, not deployed) + a design for the backup copies (Part 1.4). Architecture read: architecture/05-hub-architecture.md §16.2–§16.3 (sealing, the snapshot), 13-threat-model.md (attacker j, R-1241), audits/plex2shell-felhom-impact-2026-10-23/README.md. Names and places only — no value appears here or in any file of this session. Every count below came from a read-only copy of the hub's own 14:00Z snapshot (VACUUM INTO, so it holds every committed write), copied to a 0700 scratch folder and deleted after.

1. Every hub column that holds a secret (measured, 2026-10-23)

Where	What	Before 0.168.0	0.168.0
hosts.api_key	each box's hub key	sealed (R-879, 0.138.0) — 4/4	same
customer_configs.api_key, retrieval_password	controller key, owner passphrase	sealed (R-879) — 5/5 each	same
host_pbs_secrets.value	PBS-DR token	sealed (R-879) — 4/4	same
one_time_secrets.value	Storage Box sub-account password	sealed (R-821, 0.127.0) — 4/4	same
host_recovery.secret	break-glass console password	sealed (R-133, 0.135.0) — 4/4	same
customer_configs.config_json → infrastructure.cf_tunnel_token	Cloudflare tunnel token	plain — 4 rows	sealed
customer_configs.config_json → infrastructure.cf_api_token	Cloudflare API token	plain — 4 rows	sealed
customer_configs.config_json → git.token	git token (the form still accepts one)	none stored today	sealed when written
host_escrow.blob, identity_blob; host_escrow_superseded.*	escrow	age-wrapped by the household's recovery code — not the hub's to seal	same
api_tokens.token_hash, selfbind_tokens.token_hash, appliance_registrations.token_hash, customer_claims.code_hash, hub_settings[operator_password_hash]	operator tokens, bind tokens, claim codes, operator password	one-way hashes (SHA-256 / bcrypt)	same
hosts.api_key_hash, customer_configs.api_key_hash	lookup twins	unkeyed SHA-256 of a 256-bit random key (R-879)	same
wg_peers.pubkey, wg_endpoints.server_pubkey, host_escrow.key_fingerprint, appliance_registrations.ssh_host_pubkeys, hub_settings[oob_operator_ssh_pubkey]	public keys, fingerprints	not secret	same
guests.api_key	unused column	empty 7/7	same
appliance_registrations.pairing_code	6-character console code	plain, 31 rows	NOT-A-FINDING: it is printed on the box's screen and is only one of two factors; the other (the self-bind token) is stored as a hash. Pending registrations only.

Do the token VALUES sit anywhere else? Each of the 8 distinct Cloudflare values was searched (by value, in memory, never printed) in every column of every other table: 0 hits. The key NAME cf_tunnel_token appears in one reports.report_json and one app_telemetry.issues_json row — names, not values. events.details_json carries repo_key (literally the word „changed") and PBS token_ids (a name like user@pbs!name, not a secret).

2. The change (hub 0.168.0)

• Same mechanism, same key (05 §16.2): each token leaf in config_json holds enc:v1: + AES-256-GCM under OFFSITE_SECRET_KEY. The rest of the JSON stays readable. internal/store/r975_config_token_seal.go.
• New writes sealed: SaveCustomerConfig (the only writer of config_json). No key and a token → refused.
• Every reader opens through the existing helper: openCustomerSecrets, which the three readers (GetCustomerConfig, GetCustomerConfigByAPIKey, ListCustomerConfigs) already call. A leaf that does not open is blanked and sets SecretsUnreadable (R-879's rule: never served, a save refuses the record, configgen.Generate refuses).
• Old rows sealed once at start-up: SealLegacyConfigTokens, logged as config tokens sealed at rest (N legacy plaintext token(s) sealed now) — counts only. No config_version bump — the box's config is byte-for-byte the same, so no box re-pulls or restarts. Expected on the live hub: N = 8 on the first start, 0 after.
• Then one VACUUM + WAL checkpoint (only when N > 0): an UPDATE leaves the old plaintext row in a freed block of hub.db (the SQLite driver's secure_delete is 0 — measured). Timed on a copy of the live 534 MB database: 2.8 s, well inside the liveness probe's ~95 s. The volume has 2.4 GB free for it.
• Roll-back: felhom-hub -unseal-box-secrets now also opens these leaves (all or nothing), so going back to 0.167.0 stays the documented procedure (05 §16.2). Without it an older hub would hand the box the sealed string as its tunnel token.

Tests, each red-proved (the rule broken on purpose → the test failed → restored):

Test	Pins	Red-proof (mutation → failed)
TestR975_FreshWriteIsStoredSealed	a fresh write is sealed; non-secret fields intact; no key → refused	write path unsealed → FAIL (with 5 others)
TestR975_ReaderGetsTheSameValue	all three readers + a load-modify-save return the same value	open on read removed → FAIL
TestR975_MigrationSealsOnceThenNoOp	3 sealed, then 0; no config_version/updated_at change	migration no-op → FAIL; migration bumps the version → FAIL
TestR975_DBFileWithoutKeyRevealsNoToken	a copy of hub.db + -wal + -shm greps clean; positive control: the pre-0.168.0 rows DO show in the file	VACUUM removed → FAIL (the old row stays in a freed block)
TestR975_WrongKeyBlanksAndRefusesSave	wrong key → blanked, flagged, save refused, right key restores	not blanked → FAIL; flag ignores the tokens → FAIL
TestR975_UnsealRestoresPlaintextForAnOlderHub	roll-back opens the tokens, all or nothing	the config half skipped → FAIL
TestR975_EveryFormSecretIsSealed (web)	every leaf the operator form writes is either on a 2-name non-secret list or sealed	write path unsealed → FAIL
TestR975_GeneratedConfigCarriesThePlainTunnelToken (configgen)	the box still gets the REAL token in controller.yaml	open on read removed → FAIL

Two first mutations broke the build instead of a test (an unused variable) and were redone so a test, not the compiler, failed. The first version of the file test passed with the VACUUM removed — the token row was the last cell on its page, so the new row overwrote it; a second customer written after it (as on the live hub) made the test honest.

Run on a private copy of the real database (the 14:00Z snapshot, in a 0700 scratch folder, with a THROWAWAY key — the real one was never read; a temporary test file, deleted, never committed): the start-up sealing sealed 8 tokens in 3.15 s (incl. the VACUUM), a second run sealed 0, all 5 of 5 customers' config_json read back equal to the original (JSON-equal), and 0 of the 8 token values were found anywhere in the database files afterwards. A first attempt compared whole records and showed „0 of 5 equal": the R-879 columns were sealed with the REAL key, so with the throwaway key every record was flagged unreadable — the expected behaviour, not a defect; the second run compared the JSON only. The copy was shredded after. Image: gitea.dooplex.hu/admin/felhom-hub:0.168.0 (sha256:3bfdecaae527d4762059600cdfaad83daf4e4b5b20661fc47acf128180d87cb3), from 5277d859, CI run 2269 success.

Measured on the live file (count of the key name only): cf_tunnel_token occurs 12 times both in the live hub.db and in the clean VACUUM INTO snapshot, so the live file holds no extra old copies today.

3. The backup copies — design only (Part 1.4)

What exists today (measured):

Copy	Where	Encrypted?	Holds the tokens?
Hub snapshots <data>/snapshots/ (2, 6-hourly)	the hub's Longhorn volume	no (the volume is encrypted=false)	after 0.168.0: sealed
Longhorn backups of the hub volume	/mnt/5_hdd/backup/longhorn-pvc (the NFS folder of R-1241) — 2 backups: 2026-10-18 03:00Z (weekly) and 2026-10-23 02:01Z (daily), ~4.1 GB each, retain 1 each	no	yes, in plaintext — both predate 0.168.0
The hub's own copy to ep0 (felhom-hub-db-backup, 6-hourly :30)	ep0 PBS, namespace operator	yes, at rest: 16 snapshots 2026-10-10 → 10-23, every hubdb.pxar.didx and catalog crypt-mode: encrypt (index.json.blob sign-only). The key /etc/felhom-hub-backup/enc.key is on DooPlex's root disk (sdb1), not on the NFS disk, and on paper (S2). In transit: SSH port-forward 127.0.0.1:18007 + PBS TLS with a pinned fingerprint. Last push 14:31 CEST today, 7 s; the weekly restore-test last passed 2026-10-18 („4 sealed console password(s), 0 readable").	sealed after 0.168.0

The two options:

	(a) Encrypt the hub's Longhorn volume	(b) Take the hub volume out of the Longhorn NFS backup
What	A new Longhorn volume with encrypted=true (a StorageClass with a crypto Secret), copy /data across, point manifests/hub.yaml at it	Remove the volume from the default recurring-job group (one label on the Longhorn volume); the ep0 copy is the hub's only backup
Hub downtime	~5–10 min (scale to 0, copy 1.5 GB, scale up)	none
Who	homelab (a StorageClass + Secret in longhorn-system) + a Felhom manifest change	homelab (the Longhorn volume label); no Felhom code
New key to keep	yes — the volume passphrase, on paper, and it lives in the cluster where the R-1241 attacker reaches the Longhorn API	none
What it fixes	the backups stay but are ciphertext; also the volume's freed blocks	no more hub copies in the NFS folder at all
What it loses	nothing	the hub's 90-day access log (/data/access-log, R-1140) and its local snapshots are no longer backed up anywhere; a lost volume is restored from ep0 (RUNBOOK-hub-db-offsite-backup.md §restore, restore-tested weekly) — at most ~6.5 h of hub data lost, the same as today's off-site promise
Does the encryption key sit next to the data?	yes (cluster Secret)	the PBS key is on another disk and on paper

CC's pick: (b), because the ep0 copy is already encrypted at rest and in transit (measured above), off-site, every 6 hours, and restore-tested weekly — the Longhorn copy adds a second, unencrypted copy in exactly the folder R-1241 found open, and (a) would put its key in the same cluster. The access log's loss can be closed later by adding /data/access-log to the ep0 push (small; not tonight). → decide row R-1243.

The plain copies already in the folder (the two hub-volume backups above): they hold today's tokens in plaintext. Deleting backups is the operator's decision. Without action they rotate out by themselves after 0.168.0 is deployed — the daily one at the next 04:00 run, the weekly one at the next Sunday 05:00 run (retain 1 each) — but only if the volume stays in the backup job (option a, or no decision). With option (b) the last two copies stay until someone deletes them. Either way the tokens in them should then be considered exposed only if R-1241's door is used — the Plex2Shell audit measured it was not. Note: Longhorn backs up the volume's blocks, so a block freed by the VACUUM can still hold old plaintext in a later Longhorn backup; only (a) or (b) removes that — not measured, reasoned.