hub v0.60.1: host deletion demotes escrow custody (never destroys) + customer-delete purge point + S6b obsolete

- DeleteHost(deleteEscrow) demotes current host_escrow into host_escrow_superseded (copy-before-delete, same tx), spares existing; one shared demoteCurrentEscrowTx (reused by SaveHostEscrow). F-14 provenance/gate unchanged.
- DeleteCustomerConfig now purges both escrow tables for all the customer's hosts incl. already-deleted (F-14 provenance UNION) — the one true purge point.
- Wording: checkbox/refusal/Danger-zone → demotion. S6b OBSOLETE. Red-proofs TestDeleteHost_Demotes + TestDeleteCustomer_Purges + wording guard.
This commit is contained in:
2026-07-17 11:25:38 +02:00
parent 106c3379b0
commit 2752e12acc
12 changed files with 310 additions and 59 deletions
+30
View File
@@ -1,5 +1,35 @@
# Felhom Hub — Changelog
## v0.60.1 — host deletion DEMOTES escrow custody (never destroys) + S6b obsolete (2026-07-17)
Closes the deletion-path gap in v0.60.0's review: `DeleteHost(deleteEscrow=true)` was still DELETING
escrow rows (the same-customer reinstall flow funnels the operator straight into that tick).
Principle (Viktor's standing ruling): host deletion is a lifecycle event — blob custody survives it;
the customer Danger-zone Delete is the one true purge point. Green:
`go build ./... && go vet ./... && go test ./...`.
- **Scenario A — host delete demotes, never destroys.** `DeleteHost(deleteEscrow=true)` now DEMOTES
the current `host_escrow` row into `host_escrow_superseded` (copy-BEFORE-delete, same tx) and SPARES
existing superseded rows — no operator path through host lifecycle can lose a blob. Reuses THE one
escrow row-copy routine (`demoteCurrentEscrowTx`, also used by `SaveHostEscrow`). The F-14
provenance row + gate semantics are unchanged (wording updated: demotion, not destruction). Edge:
no escrow row → unchanged; the `ErrHostEscrowPresent` refusal without the flag is unchanged.
Red-proof `TestDeleteHost_DemotesEscrowNeverDestroys`.
- **Scenario B — customer delete is the purge point.** `DeleteCustomerConfig` (which previously deleted
ONLY the `customer_configs` row) now, in one tx, purges `host_escrow` AND `host_escrow_superseded`
for all the customer's hosts — INCLUDING already-deleted hosts (resolved via the F-14
`host_deletions` provenance) so a host-delete-then-customer-delete ordering leaves nothing orphaned.
Danger-zone copy states it. Red-proof `TestDeleteCustomer_PurgesEscrowCustody`.
- **Scenario C — wording.** The host-delete escrow checkbox now reads "Move key escrow to retained
custody (required when escrow present)…"; the refusal message + customer Danger-zone copy match.
Guard test `TestHostDeleteEscrowLabel_DemotionWording`.
- **Scenario D — S6b verdict (docs): OBSOLETE.** Re-enrolling an existing host_id upserts cleanly
(`UpsertHost` ON CONFLICT DO UPDATE, `store.go`; `handleAdminCreateHost` has no duplicate refusal) +
the v0.57.0 re-enroll arc auto-fires the re-issues → no manual stale-host deletion needed before
re-enroll. Scenario A also makes the funnel harmless either way. ROADMAP R-3 refined.
- **Scope:** hub-only; no controller/agent change; ACK assembly + upload supersede path untouched.
Deploy: bump `manifests/hub.yaml` tag to `0.60.1` and sync.
## v0.60.0 — offsite continuity Part B: superseded-escrow retention (data-first) (2026-07-17)
Closes the data-loss half of the reinstall-orphaned-repo incident: `SaveHostEscrow`'s destructive
@@ -0,0 +1,44 @@
package store
import "testing"
// Scenario B (v0.60.1) — the customer Danger-zone Delete is the one true purge point: it removes
// host_escrow AND host_escrow_superseded for ALL the customer's hosts, including hosts already deleted
// (whose demoted blobs survive as superseded rows), so nothing is left orphaned. RED-PROOF: pre-fix
// DeleteCustomerConfig deleted only customer_configs → the escrow blobs survived → the assertions FAIL.
func TestDeleteCustomer_PurgesEscrowCustody(t *testing.T) {
s := newTestStore(t)
const cust = "cust-b"
if err := s.UpsertHost(&Host{HostID: "b1", CustomerID: cust, APIKey: "k1"}); err != nil {
t.Fatal(err)
}
if err := s.UpsertHost(&Host{HostID: "b2", CustomerID: cust, APIKey: "k2"}); err != nil {
t.Fatal(err)
}
// b1: current SHA_A + superseded SHA_OLD; b2: current SHA_B.
s.SaveHostEscrow("b1", []byte("old"), "fp", "zk", "t", "SHA_OLD")
s.SaveHostEscrow("b1", []byte("A"), "fp", "zk", "t", "SHA_A")
s.SaveHostEscrow("b2", []byte("B"), "fp", "zk", "t", "SHA_B")
// ORDERING: delete b2 as a HOST first — SHA_B is demoted to a superseded row and the b2 host row is
// gone (so it can only be found again via the F-14 host_deletions provenance).
if err := s.DeleteHost("b2", true); err != nil {
t.Fatal(err)
}
if n, _ := s.CountSupersededEscrow("b2"); n != 1 {
t.Fatalf("precondition: b2 demoted blob = %d, want 1", n)
}
// The customer Danger-zone Delete purges EVERYTHING for the customer's hosts (current + orphaned).
if err := s.DeleteCustomerConfig(cust); err != nil {
t.Fatalf("DeleteCustomerConfig: %v", err)
}
for _, h := range []string{"b1", "b2"} {
if cur, _ := s.GetHostEscrow(h); cur != nil {
t.Fatalf("%s current escrow survived the customer delete", h)
}
if n, _ := s.CountSupersededEscrow(h); n != 0 {
t.Fatalf("%s retained escrow survived the customer delete: %d (orphaned)", h, n)
}
}
}
@@ -0,0 +1,70 @@
package store
import (
"errors"
"testing"
)
// Scenario A (v0.60.1) — host deletion DEMOTES the current escrow blob to retained custody and SPARES
// existing superseded blobs; it never destroys custody. RED-PROOF: the v0.60.0 code deleted both escrow
// tables → after DeleteHost the blobs are gone → the retrieval assertions FAIL.
func TestDeleteHost_DemotesEscrowNeverDestroys(t *testing.T) {
s := newTestStore(t)
const hostID, cust = "dh1", "cust-dh"
if err := s.UpsertHost(&Host{HostID: hostID, CustomerID: cust, APIKey: "k"}); err != nil {
t.Fatal(err)
}
// current escrow = SHA_A, one superseded = SHA_OLD (two uploads with different passphrases).
if _, err := s.SaveHostEscrow(hostID, []byte("blob-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
t.Fatal(err)
}
if _, err := s.SaveHostEscrow(hostID, []byte("blob-A"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
t.Fatal(err)
}
if n, _ := s.CountSupersededEscrow(hostID); n != 1 {
t.Fatalf("precondition superseded=%d, want 1", n)
}
// Refusal without the flag is unchanged (escrow present + !deleteEscrow → ErrHostEscrowPresent).
if err := s.DeleteHost(hostID, false); !errors.Is(err, ErrHostEscrowPresent) {
t.Fatalf("delete without the flag: got %v, want ErrHostEscrowPresent", err)
}
if err := s.DeleteHost(hostID, true); err != nil {
t.Fatalf("DeleteHost: %v", err)
}
if h, _ := s.GetHost(hostID); h != nil {
t.Fatal("host row survived the delete")
}
// The current row is DEMOTED (moved), not kept.
if cur, _ := s.GetHostEscrow(hostID); cur != nil {
t.Fatal("current escrow row survived — should be demoted into superseded, not left as current")
}
// RED-PROOF: BOTH blobs are RETAINED (SHA_A demoted + SHA_OLD spared) — host delete never destroys.
sup, err := s.ListSupersededEscrow(hostID)
if err != nil {
t.Fatal(err)
}
if len(sup) != 2 {
t.Fatalf("retained superseded = %d, want 2 (SHA_A demoted + SHA_OLD spared) — host delete DESTROYED custody", len(sup))
}
shas := map[string]bool{}
for _, e := range sup {
shas[e.ResticPwSHA256] = true
}
if !shas["SHA_A"] || !shas["SHA_OLD"] {
t.Fatalf("retained shas = %v, want both SHA_A + SHA_OLD", shas)
}
// EDGE: a host with NO escrow → DeleteHost unchanged (no demote row, no error).
const h2 = "dh2"
if err := s.UpsertHost(&Host{HostID: h2, CustomerID: cust, APIKey: "k2"}); err != nil {
t.Fatal(err)
}
if err := s.DeleteHost(h2, true); err != nil {
t.Fatalf("no-escrow delete: %v", err)
}
if n, _ := s.CountSupersededEscrow(h2); n != 0 {
t.Fatalf("a no-escrow host delete created a superseded row: %d", n)
}
}
+71 -23
View File
@@ -1130,10 +1130,38 @@ func (s *Store) ListCustomerConfigs() ([]CustomerConfig, error) {
return configs, rows.Err()
}
// DeleteCustomerConfig deletes a customer configuration.
// DeleteCustomerConfig deletes a customer configuration AND purges the customer's escrow custody
// (v0.60.1). The customer Danger-zone Delete is the ONE true purge point for recovery-key custody:
// host deletion only DEMOTES a blob to retained custody (never destroys), so removing the customer is
// the deliberate, acknowledged point where that retained custody is permanently removed. In one tx it
// deletes host_escrow AND host_escrow_superseded for ALL the customer's hosts — INCLUDING hosts
// already deleted (whose demoted blobs survive in host_escrow_superseded), resolved via the F-14
// host_deletions provenance so a host-delete-then-customer-delete ordering leaves nothing orphaned.
// The broader offboarding lifecycle (Hetzner sub-account, WG peer, Storage-Box data, the host rows
// themselves) is NOT this method — see the delete/re-create rehearsal (ROADMAP R-3).
func (s *Store) DeleteCustomerConfig(customerID string) error {
_, err := s.db.Exec("DELETE FROM customer_configs WHERE customer_id = ?", customerID)
return err
tx, err := s.db.Begin()
if err != nil {
return err
}
defer tx.Rollback()
// Current hosts' escrow.
if _, err := tx.Exec(`DELETE FROM host_escrow WHERE host_id IN (SELECT host_id FROM hosts WHERE customer_id = ?)`, customerID); err != nil {
return fmt.Errorf("DeleteCustomerConfig %s: purge host_escrow: %w", customerID, err)
}
// Retained (superseded) blobs for BOTH current and already-deleted hosts of this customer.
if _, err := tx.Exec(`
DELETE FROM host_escrow_superseded WHERE host_id IN (
SELECT host_id FROM hosts WHERE customer_id = ?
UNION
SELECT host_id FROM host_deletions WHERE customer_id = ?
)`, customerID, customerID); err != nil {
return fmt.Errorf("DeleteCustomerConfig %s: purge host_escrow_superseded: %w", customerID, err)
}
if _, err := tx.Exec(`DELETE FROM customer_configs WHERE customer_id = ?`, customerID); err != nil {
return fmt.Errorf("DeleteCustomerConfig %s: delete config: %w", customerID, err)
}
return tx.Commit()
}
// GetCustomerConfigByAPIKey looks up a customer config by its unique API key.
@@ -1958,8 +1986,9 @@ func (s *Store) CountHostArtifacts(hostID string) (HostArtifacts, error) {
//
// v0.53.0 (F-14 provenance): every delete also writes a host_deletions row IN THE SAME tx.
// escrow_acked = deleteEscrow AND an escrow row was actually present — "removed through the
// escrow-ack flow" means an acknowledged destruction happened, not merely that the checkbox
// was ticked over nothing.
// escrow-ack flow" means the operator acknowledged the host removal and the current escrow blob was
// DEMOTED to retained custody (v0.60.1: moved into host_escrow_superseded, not destroyed), not
// merely that the checkbox was ticked over nothing. The flag's F-14 gate semantics are unchanged.
func (s *Store) DeleteHost(hostID string, deleteEscrow bool) error {
if hostID == "" {
return fmt.Errorf("DeleteHost: empty host_id")
@@ -1997,6 +2026,19 @@ func (s *Store) DeleteHost(hostID string, deleteEscrow bool) error {
return fmt.Errorf("DeleteHost %s: customer lookup: %w", hostID, err)
}
// v0.60.1: host deletion is a LIFECYCLE event — the current escrow blob is DEMOTED to retained
// custody (copied into host_escrow_superseded, copy-BEFORE-delete in this same tx), NEVER
// destroyed; existing superseded rows are spared. No operator path through host lifecycle can
// lose a blob. The customer Danger-zone Delete is the one true purge point (deleteCustomer).
if deleteEscrow {
if _, derr := demoteCurrentEscrowTx(tx, hostID); derr != nil {
return fmt.Errorf("DeleteHost %s: demote escrow to retained custody: %w", hostID, derr)
}
if _, derr := tx.Exec(`DELETE FROM host_escrow WHERE host_id = ?`, hostID); derr != nil {
return fmt.Errorf("DeleteHost %s: remove current escrow row: %w", hostID, derr)
}
}
stmts := []string{
`DELETE FROM guests WHERE host_id = ?`,
`DELETE FROM host_reports WHERE host_id = ?`,
@@ -2006,12 +2048,8 @@ func (s *Store) DeleteHost(hostID string, deleteEscrow bool) error {
`DELETE FROM log_bundle_requests WHERE scope_id = ?`,
`DELETE FROM log_bundles WHERE scope_id = ?`,
`DELETE FROM wg_peers WHERE host_id = ?`,
`DELETE FROM hosts WHERE host_id = ?`,
}
if deleteEscrow {
stmts = append(stmts, `DELETE FROM host_escrow WHERE host_id = ?`)
stmts = append(stmts, `DELETE FROM host_escrow_superseded WHERE host_id = ?`) // retained blobs go with the host
}
stmts = append(stmts, `DELETE FROM hosts WHERE host_id = ?`)
for _, q := range stmts {
if _, err := tx.Exec(q, hostID); err != nil {
return fmt.Errorf("DeleteHost %s: %q: %w", hostID, q, err)
@@ -2021,8 +2059,8 @@ func (s *Store) DeleteHost(hostID string, deleteEscrow bool) error {
}
// HostDeletion is one host-removal provenance record (v0.53.0, F-14). EscrowAcked means the
// operator removed the host through the escrow-ack flow — an acknowledged destruction of the
// host's key custody, the ONLY state that permits the PBS-DR auto-re-issue.
// operator removed the host through the escrow-ack flow — the current key custody was DEMOTED to
// retained custody (v0.60.1), not destroyed — the ONLY state that permits the PBS-DR auto-re-issue.
type HostDeletion struct {
HostID string
CustomerID string
@@ -2095,6 +2133,23 @@ type HostEscrow struct {
// host_escrow_superseded before overwriting the current row (Part B, v0.60.0). A same-sha re-upload
// (idempotent re-ceremony of the same password) refreshes the current row and does NOT create a
// superseded row.
// demoteCurrentEscrowTx copies the host's CURRENT host_escrow row (if any) into
// host_escrow_superseded as a retained blob, inside the given tx. This is THE ONE escrow row-copy
// routine (v0.60.0): SaveHostEscrow uses it to retain a superseded different-passphrase blob before
// overwriting, and DeleteHost (v0.60.1) uses it to DEMOTE the current blob to retained custody
// instead of destroying it. Returns the number of rows copied (0 when the host has no current row).
// The hub never decrypts; custody is unchanged.
func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
res, err := tx.Exec(`
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
FROM host_escrow WHERE host_id = ?`, hostID)
if err != nil {
return 0, err
}
return res.RowsAffected()
}
func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, err error) {
tx, err := s.db.Begin()
if err != nil {
@@ -2108,13 +2163,9 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
// Retain the current row iff it exists AND seals a DIFFERENT restic password (the incident: a
// recreated volume mints a new passphrase; the old must stay recoverable with its recovery code).
var (
curBlob []byte
curFp, curPosture, curCreated, curSHA string
exists bool
)
row := tx.QueryRow(`SELECT blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256,'') FROM host_escrow WHERE host_id = ?`, hostID)
switch scanErr := row.Scan(&curBlob, &curFp, &curPosture, &curCreated, &curSHA); scanErr {
var curSHA string
var exists bool
switch scanErr := tx.QueryRow(`SELECT COALESCE(restic_pw_sha256,'') FROM host_escrow WHERE host_id = ?`, hostID).Scan(&curSHA); scanErr {
case nil:
exists = true
case sql.ErrNoRows:
@@ -2124,10 +2175,7 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
return false, err
}
if exists && curSHA != resticPwSHA256 {
if _, err = tx.Exec(`
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
VALUES (?, ?, ?, ?, ?, ?, datetime('now'))`,
hostID, curBlob, curFp, curPosture, curCreated, curSHA); err != nil {
if _, err = demoteCurrentEscrowTx(tx, hostID); err != nil {
return false, err
}
superseded = true
@@ -0,0 +1,30 @@
package web
import (
"strings"
"testing"
)
// Scenario C (v0.60.1) — the host-delete escrow checkbox must tell the truth: host deletion DEMOTES
// the blob to retained custody, it does not destroy it. Guards the wording against a regression.
func TestHostDeleteEscrowLabel_DemotionWording(t *testing.T) {
b, err := templateFS.ReadFile("templates/host_detail_body.html")
if err != nil {
t.Fatal(err)
}
src := string(b)
if !strings.Contains(src, "Move key escrow to retained custody") {
t.Error("host-delete escrow checkbox must use demotion wording (retained custody)")
}
if strings.Contains(src, "Also delete the key escrow") {
t.Error("host-delete escrow checkbox still uses destruction wording (regression)")
}
// The customer Danger-zone copy names the permanent purge of retained custody (the one true point).
cb, err := templateFS.ReadFile("templates/customer_unified.html")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(cb), "retained recovery-key custody") {
t.Error("customer Danger-zone copy must name the permanent removal of retained recovery-key custody")
}
}
+1 -1
View File
@@ -485,7 +485,7 @@ func (s *Server) handleHostDelete(w http.ResponseWriter, r *http.Request, hostID
if err := s.store.DeleteHost(hostID, deleteEscrow); err != nil {
if errors.Is(err, store.ErrHostEscrowPresent) {
s.logger.Printf("[WARN] host delete refused: %s has key escrow (acknowledgement missing)", hostID)
http.Error(w, "This host has a key escrow (+ DR bundle). Tick the escrow acknowledgement to delete it too — nothing deleted.", http.StatusConflict)
http.Error(w, "This host has a key escrow (+ DR bundle). Tick the escrow acknowledgement to move it to retained custody — nothing deleted.", http.StatusConflict)
return
}
s.logger.Printf("[ERROR] host delete %s: %v", hostID, err)
@@ -706,7 +706,7 @@
Customer Info header — endpoints and confirm() handlers unchanged. -->
<section class="card">
<h2>Danger zone</h2>
<p class="text-muted">Blocking hides the customer from the Dashboard (reports are still accepted); deleting removes the managed configuration permanently.</p>
<p class="text-muted">Blocking hides the customer from the Dashboard (reports are still accepted); deleting removes the managed configuration permanently — and permanently removes the retained recovery-key custody (escrow blobs) for this customer's hosts. This is the one true purge point; host deletion only demotes custody, never destroys it.</p>
<div style="display: flex; gap: 0.5rem; flex-wrap: wrap; margin-top: 0.5rem;">
{{if .IsBlocked}}
<form method="POST" action="/customers/{{.CustomerID}}/unblock" style="display:inline">
@@ -284,7 +284,7 @@
<p id="host-delete-impact-{{.HostID}}" style="margin: 0 0 0.5rem; font-size: 0.9em;">&hellip;</p>
<label id="host-delete-escrow-row-{{.HostID}}" style="display: none; margin: 0 0 0.5rem; font-size: 0.85em;">
<input type="checkbox" id="host-delete-escrow-{{.HostID}}">
Also delete the key escrow + DR bundle for this host
Move key escrow to retained custody (required when escrow present) + remove DR bundle for this host
</label>
<p style="margin: 0 0 0.5rem; font-size: 0.85em; color: var(--text-2);">Type the host id to confirm:</p>
<form method="POST" action="/hosts/{{.HostID}}/delete" id="host-delete-form-{{.HostID}}" style="display: flex; gap: 0.5rem; align-items: center; flex-wrap: wrap;">