admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196)
gates / gates (push) Successful in 7s
Details

Browse Source 
R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and
identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY
password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE
ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site
data key, which is the secret the retention was built to preserve. And because the copy happens
as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing
a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups
stay recoverable. Both demo boxes crossed that line on 2026-08-04.

  - identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy
    routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion.
  - ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob.
  - CountCurrentEscrowWithIdentity is the census of who the fix protects.
  - Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone.
  - Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why
    the pre-existing retention test stayed green for two months asserting the mechanism.

R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises
offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and
differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity
blob retained, a changed key is "this history now depends on an older recovery code", not a loss.

R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and
regressed, each stating its own measurement; the regressed text withdraws the Re-issue
recommendation. Every self-heal refusal leaves a notification_log row with its reason. The
guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in
the alert text so the limitation travels with the number. offsite_delivery_stuck and
offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither
has a customerMessages entry, which is not a block.

R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the
restic repo password. It resets the PROVIDER password and cannot touch the repo password, which
is generated on the box. All five corrected; the staleness mark documented as precautionary. The
BEHAVIOUR stays open.

Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is
still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).
This commit is contained in:
authentik Default Admin
2026-08-04 12:56:58 +02:00
parent 3f2b7bc023
commit 91cabdde1b
28 changed files with 1122 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hub/internal/store/store.go
+91 -21
View File
@@ -374,6 +374,22 @@ func (s *Store) migrate() error {
-- "history destroyed" into "history recoverable with the recovery code"). Append-only; the hub
-- never decrypts; NO pruning (the blobs are tiny + R-encrypted; custody unchanged). The ACK and
-- restore-serving read host_escrow (the CURRENT row) — never this table.
--
-- THE RULING ABOVE WAS NOT MET FOR TWO MONTHS, AND THIS IS THE RECORD OF IT (R-198, fixed
-- v0.93.0). This table shipped with the blob column — the K-escrow, i.e. the PBS datastore key
-- — and identity_blob was added to host_escrow LATER (the slice-10D ALTER below), never here.
-- The offsite restic REPOSITORY password lives in identity_blob, not in blob. So the retention
-- preserved the whole-guest key and silently dropped the off-site data key: precisely the
-- secret the reinstall-orphan incident was about. Worse, the copy happens as the new blob
-- overwrites the old, so the destroying act was the ESCROW CEREMONY — the exact thing a
-- rebuilt box asks its customer to run, on a card promising the old backups stay recoverable.
-- Both demo boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08) and their previous
-- repository passwords are unrecoverable, recovery code or not.
-- identity_blob is now carried (see demoteCurrentEscrowTx, which is still THE ONE row-copy
-- routine). Pinned by TestSaveHostEscrow_RetainsIdentityBlob and
-- TestDeleteHost_DemotesIdentityBlob — the routine is proven through BOTH of its callers,
-- because a shared routine tested through one caller is how a fix gets believed on a path
-- nobody exercised. Evidence: audits/RECON-offsite-dr-chain-2026-08-04.md §7.
CREATE TABLE IF NOT EXISTS host_escrow_superseded (
id INTEGER PRIMARY KEY AUTOINCREMENT,
host_id TEXT NOT NULL,
@@ -382,7 +398,8 @@ func (s *Store) migrate() error {
posture TEXT NOT NULL DEFAULT '',
created_at DATETIME NOT NULL,
restic_pw_sha256 TEXT NOT NULL DEFAULT '',
superseded_at DATETIME NOT NULL DEFAULT (datetime('now'))
superseded_at DATETIME NOT NULL DEFAULT (datetime('now')),
identity_blob BLOB
);
CREATE INDEX IF NOT EXISTS idx_host_escrow_superseded_host ON host_escrow_superseded(host_id);
@@ -414,16 +431,28 @@ func (s *Store) migrate() error {
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN identity_blob BLOB`)
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN directive_json TEXT NOT NULL DEFAULT '{}'`)
// R-198 (v0.93.0) — the SAME column on the RETAINED table. It was added above and not here, and
// that omission is what made the retention keep the wrong key for two months (see the comment on
// host_escrow_superseded). Additive and tolerated on re-run, exactly like the lines above; it
// changes no existing row. Rows superseded BEFORE this ships were written without the identity
// blob and their source rows are already overwritten — there is nothing to backfill, and the
// v0.93.0 report records that as a looked-at fact rather than a deduction.
s.db.Exec(`ALTER TABLE host_escrow_superseded ADD COLUMN identity_blob BLOB`)
// SLICE 3 (escrow auto-confirm) — sha256 hex of the offsite restic repo password sealed in the
// identity blob. The hash of a 256-bit random secret is non-reversible/non-brute-forceable — safe to
// store and serve; it lets the controller VERIFY "the escrow covers the CURRENT repo password"
// instead of trusting blob-presence. NULL/'' = a legacy or password-less blob (never auto-confirms).
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN restic_pw_sha256 TEXT`)
// v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at is set when the offsite repo
// password is re-issued: the blob then seals a password that no longer opens the repo, so the
// hub must stop advertising "ceremony done" and withhold the (now non-matching) restic_pw_sha256
// from the auto-confirm ACK. NULL = current; a fresh ceremony (SaveHostEscrow) clears it.
// v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at is set when the offsite PROVIDER
// credentials are re-issued. ⚠ CORRECTED 2026-08-04 (R-196): it used to say "when the offsite repo
// password is re-issued", which nothing in the hub does — the repository password is generated on
// the box and never leaves it except sealed under R. The flag is PRECAUTIONARY (the box's re-apply
// MAY mint a fresh repository password), not evidence that it changed; the evidential signal is
// R-197's offsite_repo_key_changed. While set, the hub stops advertising "ceremony done" and
// withholds the possibly-non-matching restic_pw_sha256 from the auto-confirm ACK. NULL = current;
// a fresh ceremony (SaveHostEscrow) clears it.
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN stale_at DATETIME`)
// dr_recipe (SPIKE-dr-recipe-2026-06-16): the secret-free DR reconstruction recipe, stored
@@ -2529,6 +2558,11 @@ type HostEscrow struct {
// StaleAt (v0.57.0, 2.3) — non-empty when the offsite password was re-issued after this blob was
// sealed: the blob is stale (seals a password that no longer opens the repo). Cleared by a fresh ceremony.
StaleAt string
// IdentityBlob (R-198, v0.93.0) — the age-wrapped identity bundle, which is where the offsite
// restic REPOSITORY password lives. Populated by ListSupersededEscrow so a retained blob is
// reachable from Go at all; nil for pre-v0.93.0 retained rows and for hosts that never uploaded
// one. Opaque: useless without the customer's recovery code, which the hub never holds.
IdentityBlob []byte
}
// SaveHostEscrow stores (last-write-wins) the OPAQUE escrow blob for a host. The hub keeps the
@@ -2538,16 +2572,35 @@ type HostEscrow struct {
// host_escrow_superseded before overwriting the current row (Part B, v0.60.0). A same-sha re-upload
// (idempotent re-ceremony of the same password) refreshes the current row and does NOT create a
// superseded row.
//
// R-197 (v0.93.0): it also returns prevResticPwSHA256 — the hash the row being replaced sealed ("" when
// no row existed, or when it was a legacy hash-less blob). Both halves of "did the box's offsite DATA
// key change?" have been in this database since SLICE 3 and NOTHING compared them; demo-felhom's key
// changed on 2026-08-03 and no signal of any kind fired for thirteen hours. The comparison is the
// caller's (api.handleHostEscrowPut) because the event needs the customer id; returning the value is
// this function's part. The VALUE is a non-reversible hash of a 256-bit random secret and is never
// logged, mailed or written to a report — see the caller.
// demoteCurrentEscrowTx copies the host's CURRENT host_escrow row (if any) into
// host_escrow_superseded as a retained blob, inside the given tx. This is THE ONE escrow row-copy
// routine (v0.60.0): SaveHostEscrow uses it to retain a superseded different-passphrase blob before
// overwriting, and DeleteHost (v0.60.1) uses it to DEMOTE the current blob to retained custody
// instead of destroying it. Returns the number of rows copied (0 when the host has no current row).
// The hub never decrypts; custody is unchanged.
//
// R-198 (v0.93.0): `identity_blob` is copied too. It was omitted from this SELECT for two months, so
// every supersession retained the PBS datastore key and destroyed the offsite restic repository
// password — the one secret the retention exists to preserve. Copying more opaque bytes gains the hub
// NO knowledge: it still has no recovery code and no decrypt path.
//
// ORDERING THIS DEPENDS ON, stated because it is load-bearing and invisible from here: the identity
// blob is written by SaveHostDRBundle AFTER SaveHostEscrow returns (api/handler.go, the escrow PUT),
// so at demote time host_escrow still holds the OLD identity blob. If that order ever changes, this
// routine silently retains the NEW blob under the OLD blob's hash — pinned by
// TestSaveHostEscrow_RetainsIdentityBlob, which asserts the retained bytes are the previous ones.
func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
res, err := tx.Exec(`
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at, identity_blob)
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now'), identity_blob
FROM host_escrow WHERE host_id = ?`, hostID)
if err != nil {
return 0, err
@@ -2555,10 +2608,10 @@ func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
return res.RowsAffected()
}
func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, err error) {
func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, prevResticPwSHA256 string, err error) {
tx, err := s.db.Begin()
if err != nil {
return false, err
return false, "", err
}
defer func() {
if err != nil {
@@ -2577,11 +2630,14 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
exists = false
default:
err = scanErr
return false, err
return false, "", err
}
if exists {
prevResticPwSHA256 = curSHA // R-197: the caller compares; "" = no row or a legacy hash-less blob
}
if exists && curSHA != resticPwSHA256 {
if _, err = demoteCurrentEscrowTx(tx, hostID); err != nil {
return false, err
return false, prevResticPwSHA256, err
}
superseded = true
}
@@ -2598,10 +2654,10 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
stale_at = NULL,
updated_at = datetime('now')`,
hostID, blob, keyFingerprint, posture, createdAt, resticPwSHA256); err != nil {
return false, err
return false, prevResticPwSHA256, err
}
err = tx.Commit()
return superseded, err
return superseded, prevResticPwSHA256, err
}
// CountSupersededEscrow returns how many retained (superseded) escrow blobs the hub holds for a host
@@ -2616,7 +2672,7 @@ func (s *Store) CountSupersededEscrow(hostID string) (int, error) {
// first. Opaque bytes — the hub never decrypts. Seeds the future guided-recovery flow (R-26).
func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
rows, err := s.db.Query(`
SELECT host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at
SELECT host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at, identity_blob
FROM host_escrow_superseded WHERE host_id = ? ORDER BY id DESC`, hostID)
if err != nil {
return nil, err
@@ -2625,7 +2681,7 @@ func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
var out []HostEscrow
for rows.Next() {
var e HostEscrow
if err := rows.Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.ResticPwSHA256, &e.UpdatedAt); err != nil {
if err := rows.Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.ResticPwSHA256, &e.UpdatedAt, &e.IdentityBlob); err != nil {
return nil, err
}
out = append(out, e)
@@ -2633,10 +2689,22 @@ func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
return out, rows.Err()
}
// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3) — called when the offsite repo
// password is re-issued, because the blob then seals a password that no longer opens the repo. No-op
// when no escrow row exists; idempotent (only stamps the first re-issue since the last ceremony; a
// fresh ceremony clears stale_at via SaveHostEscrow's ON CONFLICT).
// CountCurrentEscrowWithIdentity returns how many hosts hold a CURRENT escrow row, and how many of
// those carry an identity blob — i.e. the population whose offsite repository password R-198's fix
// now protects from the next ceremony, and the remainder for whom there is nothing to protect
// because no identity blob was ever uploaded. Read-only; no blob or hash leaves this call.
func (s *Store) CountCurrentEscrowWithIdentity() (total, withIdentity int, err error) {
err = s.db.QueryRow(`
SELECT COUNT(*), COALESCE(SUM(identity_blob IS NOT NULL), 0) FROM host_escrow`).Scan(&total, &withIdentity)
return total, withIdentity, err
}
// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3) — called when the offsite
// PROVIDER credentials are re-issued. ⚠ CORRECTED 2026-08-04 (R-196): it used to say "when the offsite
// repo password is re-issued", which no hub path does; see the reasoning at offsite.ReissueCredentials.
// The flag is precautionary — the box's re-apply may mint a fresh repository password — not a
// measurement that one did. No-op when no escrow row exists; idempotent (only stamps the first
// re-issue since the last ceremony; a fresh ceremony clears stale_at via SaveHostEscrow's ON CONFLICT).
func (s *Store) MarkEscrowStale(hostID string) error {
_, err := s.db.Exec(`UPDATE host_escrow SET stale_at = datetime('now') WHERE host_id = ? AND stale_at IS NULL`, hostID)
return err
@@ -2665,9 +2733,11 @@ type EscrowStatus struct {
IdentityBlobPresent bool `json:"identity_blob_present"`
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
CreatedAt string `json:"created_at,omitempty"`
// Stale (v0.57.0, 2.3) — true when the offsite password was re-issued after the blob was sealed.
// Stale (v0.57.0, 2.3) — true when the offsite PROVIDER credentials were re-issued after the blob
// was sealed. ⚠ CORRECTED 2026-08-04 (R-196): it used to say "the offsite password was re-issued",
// which reads as the repository password and is not what happens; see MarkEscrowStale.
// When stale the ResticPwSHA256 is WITHHELD (emptied) so the controller cannot auto-confirm against
// a hash that no longer matches the live repo password — the ceremony must run again.
// a hash that may no longer match the live repo password — the ceremony must run again.
Stale bool `json:"escrow_stale,omitempty"`
}