diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index 9e0e79d0..f28533d7 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -454,7 +454,7 @@ or a household. The ruling: a delete removes them, with one exception — the de | `offsite_large_requests` (the hub's own first sight of a large removed-app request; `07` §8.2f — built, not released) | the household | the household is deleted; the daily orphan sweep | `store.PurgeCustomerResidue`, `store.PurgeOrphanRows`. At most 50 rows per household | | `offsite_app_presence` (the hub's own record of which apps a box runs; D72, `07` §8.2f — built, not released) | the household | the household is deleted; the daily orphan sweep takes what an earlier delete left | `store.PurgeCustomerResidue`, `store.PurgeOrphanRows`. At most 400 names per household; no age limit on a row (it would move a real app's 30 days) | | `operator_actions` (the operator's own action log) | the household | **one year after the deletion**, with `events` and `notification_log` | `store.PruneDeletedCustomerAudit`. *Decided by CC unattended on 2026-11-14; accepted by the operator on 2026-11-15 (`09` §3 decision 311):* it is an audit record of the same kind as the other two, and the one-year rule (decision 181) is the design's rule for those; the other option, at once, loses „who did what” on the day of the deletion | -| **`host_deletions`, `customer_resets`** — the delete journal | both | **never** | the ruling's one exception: an id and timestamps. **An id can be a name, so this is not „no personal data”** — the design to change that is §14.3, and the privacy text waits for it (`09` §3 decision 312) | +| **`host_deletions`, `customer_resets`** — the delete journal | both | **never** | the ROWS stay for ever; **their ids go one year after the household's deletion** (§14.3, D86 — built, not released). Until that release the ids stay, and an id can be a name | **Rows left by boxes and households deleted before this release** (Peti's box, the test boxes of 2026-11): the daily prune runs `store.PurgeOrphanRows` — the six tables of the first and third rows, for a box with no `hosts` row and for @@ -469,7 +469,7 @@ its own design, because the kernel lane reads a box's old reports to know which (`TestD73_*`), 16 mutations seen red (`audits/night-build-2026-11-14/B/redproof-hub-d73.txt`). -### 14.3 The delete journal keeps an id that is not a name `[DESIGN — proposed, not built]` (`09` §3 decision 312; written 2026-11-15) +### 14.3 The delete journal keeps no id after one year `[FACT — built, not released: hub `main` after v0.182.0, 2026-11-15]` (D86 = option A, `09` §3 decisions 312 and 314; designed 2026-11-15) **Why.** §14.2 says the delete journal stays for ever and holds „an id and timestamps, no personal data”. The second half is not true. An id of a household or of a box is typed by a person and can be a name: one deleted box carried @@ -531,8 +531,39 @@ thing the sentence must not promise away. holds its id or its boxes' ids (searched as text, with a look-alike id that must stay); the rows are still there with their times; a household deleted 364 days ago keeps its id; a household that still exists keeps everything. -**If nothing is decided:** the journal keeps ids that can be names with no end, and the privacy sentence stays -unpublished. +**Ruled 2026-11-15 evening (D86 = remove, option A).** + +**As built (`store.PruneDeletedCustomerAudit`, `pruneOneDeletedCustomer`).** At a deleted household's year-end, in ONE +transaction: its events, mail log and action log are deleted, and the journal rows of that deleted life lose their ids +— `customer_resets.customer_id`, `host_deletions.customer_id` and `host_deletions.host_id` become the EMPTY string. +The rows, their times and their step states stay. + +Four things the design above did not say, found by reading it against a broken and a hostile case before the build: + +| Case | What is built | Why | +|---|---|---| +| The placeholder | the empty string, never a word („removed”) | a word can be typed as a new household's id; the next day's run would then take that household for a deleted one and delete its log. An id is never empty | +| An id that is used AGAIN for a new household (the test household was deleted and made again three times in 2026-11) | only the rows of the DELETED life are emptied: journal rows started, and box deletes made, at or before that deletion's completion | the new household's box deletes and RESETs are read by live code (the key-package carry, a later delete's purge) | +| The emptied row at the next daily run | the run skips rows with no id | it would otherwise run the year-end for the household „” every day | +| `legs_json` | at the year-end every key and value that is not a plain lower-case word becomes „other” | rule 5 asked for a test that the field holds state words only; a test cannot bind a later writer, the year-end can | + +| Rule above | Test (`hub/internal/store/deleted_customer_audit_test.go`) | +|---|---| +| after the year no journal row holds the household's or its box's id; the rows and times stay; a look-alike id, a household 364 days after its delete, a household whose delete never ended and a household that exists keep everything | `TestD86_AfterTheYearTheJournalHoldsNoIDAndKeepsItsRows` | +| an id used again keeps the journal of its new life | `TestD86_AnIDUsedAgainKeepsTheJournalOfItsNewLife` | +| the emptied row is never read as a household; no word in place of the id | `TestD86_TheEmptiedRowIsNeverReadAsAHousehold` | +| the step field holds plain words only after the year | `TestD86_TheStepFieldKeepsOnlyPlainWords` | +| the log and the id go in one transaction (rule 2) | `TestD86_TheLogAndTheIDGoInOneTransaction` | + +15 mutations seen red, one of them only after the test was made to ask for the staged error +(`audits/build-release-4b-2026-11-15/C/redproof-hub-journal-id.txt`). **Not proven:** on a deployed hub — and no +household's year ends before 2027-09-25, so the first real run is then. + +**Rule 4 (a household whose delete never ended) is NOT built as a page line:** such a household keeps its ids (its +year has not started), and nothing on the System page names it — R-1403. + +**The privacy sentence is still not published** (decision 314): it goes to the operator with the lawyer rows +(R-802, R-813). After this build the sentence above is true for a hub that runs it. ## 15. Customer e-mails — what the hub writes to a household [hub v0.118.0, R-558] diff --git a/documentation/audits/build-release-4b-2026-11-15/C/redproof-hub-journal-id.txt b/documentation/audits/build-release-4b-2026-11-15/C/redproof-hub-journal-id.txt new file mode 100644 index 00000000..9cfbd37d --- /dev/null +++ b/documentation/audits/build-release-4b-2026-11-15/C/redproof-hub-journal-id.txt @@ -0,0 +1,65 @@ +== MUTATION: the journal keeps the household's id for ever (no update of customer_resets) + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_AfterTheYearTheJournalHoldsNoIDAndKeepsItsRows (0.04s) + deleted_customer_audit_test.go:140: PruneDeletedCustomerAudit peti: customer_resets: missing argument with index 3 +== MUTATION: the box's id stays in host_deletions + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_AfterTheYearTheJournalHoldsNoIDAndKeepsItsRows (0.05s) + deleted_customer_audit_test.go:143: a year after the deletion 1 journal row(s) still hold the household's id or its box's id +== MUTATION: the journal ROWS are deleted, not emptied + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_AfterTheYearTheJournalHoldsNoIDAndKeepsItsRows (0.04s) + deleted_customer_audit_test.go:146: the journal lost rows: 8 before, 7 after — the id goes, the row stays +== MUTATION: the new life's box deletes lose their ids too (no date limit on host_deletions) + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_AnIDUsedAgainKeepsTheJournalOfItsNewLife (0.05s) + deleted_customer_audit_test.go:182: the NEW life's box delete lost its id (0) — the escrow carry of the living household reads it +== MUTATION: the new life's RESET rows lose their ids too + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_AnIDUsedAgainKeepsTheJournalOfItsNewLife (0.05s) + deleted_customer_audit_test.go:185: the NEW life's RESET row: 0 with the id, want 1 +== MUTATION: a word in place of the empty string + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_TheEmptiedRowIsNeverReadAsAHousehold (0.04s) + deleted_customer_audit_test.go:205: the next day's run deleted 1 event row(s) of a household it took the emptied journal row for (err ) +== MUTATION: the emptied row is read as a household again + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_TheEmptiedRowIsNeverReadAsAHousehold (0.06s) + deleted_customer_audit_test.go:205: the next day's run deleted 1 event row(s) of a household it took the emptied journal row for (err ) +== MUTATION: the step field is kept as it is + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_TheStepFieldKeepsOnlyPlainWords (0.05s) + deleted_customer_audit_test.go:224: legs_json after the year = {"hosts":"ok","hetzner":"sub-account kata.kovacs@example.com removed","customer_delete":"ok","Kata Kovacs":"ok"} +== MUTATION: a value that is not a plain word stays in the step field + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_TheStepFieldKeepsOnlyPlainWords (0.04s) + deleted_customer_audit_test.go:224: legs_json after the year = {"customer_delete":"ok","hetzner":"sub-account kata.kovacs@example.com removed","hosts":"ok","other":"ok"} +== MUTATION: a key that is not a plain word stays in the step field + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_TheStepFieldKeepsOnlyPlainWords (0.04s) + deleted_customer_audit_test.go:224: legs_json after the year = {"Kata Kovacs":"ok","customer_delete":"ok","hetzner":"other","hosts":"ok"} +== MUTATION: the audit rows are deleted outside the transaction + *** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0) +== MUTATION: a household one day short of the year is emptied + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_AfterTheYearTheJournalHoldsNoIDAndKeepsItsRows (0.04s) + deleted_customer_audit_test.go:152: the step states of the deleted household's row are not kept as they were: 2 +== MUTATION: a household whose delete never ended is emptied + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_AfterTheYearTheJournalHoldsNoIDAndKeepsItsRows (0.04s) + deleted_customer_audit_test.go:156: peti-2: 0 journal row(s) hold the id after the run, want 2 (nothing of it may change) + deleted_customer_audit_test.go:156: living: 0 journal row(s) hold the id after the run, want 2 (nothing of it may change) + +13 mutations, 1 not a red-proof + +## redone, and one more +== MUTATION: the audit rows are deleted outside the transaction (redone: the mutant failed on a database lock before the journal step, and the test took any error; it asks for the staged one now) + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_TheLogAndTheIDGoInOneTransaction (4.96s) + deleted_customer_audit_test.go:238: the staged failure of the journal step was not returned: PruneDeletedCustomerAudit mari: events: database is locked (5) (SQLITE_BUSY) +== MUTATION: the journal step's failure is swallowed and the audit rows go + RED on the mutant, GREEN on the real file + --- FAIL: TestD86_TheLogAndTheIDGoInOneTransaction (0.04s) + deleted_customer_audit_test.go:238: the staged failure of the journal step was not returned: + +2 mutations, 0 not a red-proof diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 20ce374e..aa6aeb1c 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,12 @@ ## Unreleased (after v0.182.0) +- **D86 (2026-11-15 evening, `09` §3 decision 314; `05` §14.3): the delete journal keeps no id after one year.** + `store.PruneDeletedCustomerAudit` now runs each deleted household's year-end in ONE transaction and, with the audit + rows, empties the ids of that deleted life's journal rows (`customer_resets.customer_id`, + `host_deletions.customer_id` and `.host_id` → the empty string; the rows, times and step states stay). Only rows of + the deleted life (an id used again keeps its new rows); a row with no id is skipped by the next run; `legs_json` + keeps plain words only. Tests `TestD86_*` (5); two older tests counted journal rows by id and now count the rows; + 15 mutations seen red. The first real run is on 2027-09-25. - **D85 (2026-11-15, R-1379; `08` §12.3): the hub reads the end dates of its own public names once a day.** `monitor.OwnNameChecker` + `monitor.DialCertEnd`; daily job `own-names-check` at 06:40 Budapest, watched by the daily ledger. 14 days before a name's certificate ends: `hub_name_cert_ends_soon` (warning; error once the date passed and diff --git a/hub/internal/store/d73_gone_rows_test.go b/hub/internal/store/d73_gone_rows_test.go index 3507d12d..d25543c7 100644 --- a/hub/internal/store/d73_gone_rows_test.go +++ b/hub/internal/store/d73_gone_rows_test.go @@ -185,9 +185,10 @@ func TestD73_PurgeRemovesTheHouseholdsRows_JournalStays(t *testing.T) { if n := d73Counts(t, s, "box-cust-b", "cust-b")["operator_actions"]; n != 1 { t.Errorf("the OTHER household's action log: %d, want 1", n) } + // the journal rows stay; since D86 (`05` §14.3) the deleted household's rows hold no id after the year for _, j := range []string{"customer_resets", "host_deletions"} { - if n := d73Count(t, s, `SELECT COUNT(*) FROM `+j+` WHERE customer_id = 'cust-a'`); n != 1 { - t.Errorf("journal %s after the one-year prune: %d row(s), want 1", j, n) + if n := d73Count(t, s, `SELECT COUNT(*) FROM `+j+` WHERE customer_id = ''`); n != 1 { + t.Errorf("journal %s after the one-year prune: %d row(s) with no id, want 1 (the row stays, the id goes)", j, n) } } } diff --git a/hub/internal/store/deleted_customer_audit.go b/hub/internal/store/deleted_customer_audit.go index 191ab15b..5c5d4913 100644 --- a/hub/internal/store/deleted_customer_audit.go +++ b/hub/internal/store/deleted_customer_audit.go @@ -3,6 +3,7 @@ package store import ( "encoding/json" "fmt" + "regexp" "time" ) @@ -17,15 +18,57 @@ import ( // WHICH customers: the ones whose delete cascade COMPLETED — a `customer_resets` journal row with leg // `customer_delete` = `ok` and a `completed_at` stamp (a RESET has no such leg and is not a deletion). WHICH rows: only // those created AT OR BEFORE that completion — a customer id re-used after the deletion keeps everything it made since. -// The journal row itself stays (F-14 provenance: an id and timestamps, no personal data). +// The journal row itself stays (F-14 provenance) — and since D86 it loses its id in the same step (below). // // Pinned by TestR901_* (deleted_customer_audit_test.go). const DeletedCustomerAuditKeep = 365 * 24 * time.Hour -// PruneDeletedCustomerAudit deletes the events and notification_log rows of customers whose deletion completed before -// now-keep. Returns the rows deleted per table. +// D86 (operator ruling 2026-11-15, `09` §3 decision 314; `05-hub-architecture.md` §14.3, option A): an id of a household +// or a box is typed by a person and can be a name, so the delete journal must not keep it for ever. Every reader of the +// journal needs the id only while the household exists or during the year after its deletion; so, in the SAME +// transaction that removes a deleted household's audit rows at the year's end, the journal rows of that deleted life +// lose their ids: `customer_resets.customer_id`, `host_deletions.customer_id` and `host_deletions.host_id` become the +// empty string. The rows stay, with their times and their step states — „a deletion happened, at this time”. +// +// - The empty string, never a word: a word could be typed as a new household's id, and the daily run would then +// treat that household's rows as a deleted one's. An id is never empty, so no reader can match these rows. +// - Only the rows of the DELETED LIFE: those started (or, for a box, deleted) at or before that deletion's +// completion. An id used again for a new household keeps every journal row it made since — the new household's +// box deletes and RESETs are read by live code (the escrow carry, the purge of a later delete). +// - `legs_json` keeps its step names and state words; anything in it that is not a plain lower-case word is replaced +// by „other”, so a later writer cannot leave a name there. +// +// Pinned by TestD86_* (deleted_customer_audit_test.go). +const journalIDRemoved = "" + +var journalWordRe = regexp.MustCompile(`^[a-z][a-z0-9_]{0,39}$`) + +// scrubLegs returns legs_json with every key and value that is not a plain word replaced by "other". +func scrubLegs(raw string) string { + m := map[string]string{} + if json.Unmarshal([]byte(raw), &m) != nil { + return "{}" + } + out := map[string]string{} + for k, v := range m { + if !journalWordRe.MatchString(k) { + k = "other" + } + if !journalWordRe.MatchString(v) { + v = "other" + } + out[k] = v + } + b, _ := json.Marshal(out) + return string(b) +} + +// PruneDeletedCustomerAudit deletes the events, notification_log and operator_actions rows of customers whose deletion +// completed before now-keep, and in the same transaction removes the ids from that deleted life's journal rows (D86). +// Returns the audit rows deleted per table. A household whose delete never completed is not touched: its year has +// not started. func (s *Store) PruneDeletedCustomerAudit(now time.Time, keep time.Duration) (events, notifications int64, err error) { - rows, err := s.db.Query(`SELECT customer_id, completed_at, legs_json FROM customer_resets WHERE completed_at IS NOT NULL`) + rows, err := s.db.Query(`SELECT customer_id, completed_at, legs_json FROM customer_resets WHERE completed_at IS NOT NULL AND customer_id != ''`) if err != nil { return 0, 0, err } @@ -52,24 +95,69 @@ func (s *Store) PruneDeletedCustomerAudit(now time.Time, keep time.Duration) (ev } rows.Close() for _, d := range dels { - cutoff := d.at.Format("2006-01-02 15:04:05") - r, err := s.db.Exec(`DELETE FROM events WHERE customer_id = ? AND created_at <= ?`, d.id, cutoff) + ev, nl, err := s.pruneOneDeletedCustomer(d.id, d.at.Format("2006-01-02 15:04:05")) if err != nil { - return events, notifications, fmt.Errorf("PruneDeletedCustomerAudit %s: events: %w", d.id, err) - } - n, _ := r.RowsAffected() - events += n - r, err = s.db.Exec(`DELETE FROM notification_log WHERE customer_id = ? AND created_at <= ?`, d.id, cutoff) - if err != nil { - return events, notifications, fmt.Errorf("PruneDeletedCustomerAudit %s: notification_log: %w", d.id, err) - } - n, _ = r.RowsAffected() - notifications += n - // D73 (R-1376): the operator's action log of the deleted household is an audit record of the same kind — it had - // no delete at all; it goes with the other two. - if _, err = s.db.Exec(`DELETE FROM operator_actions WHERE customer_id = ? AND requested_at <= ?`, d.id, cutoff); err != nil { - return events, notifications, fmt.Errorf("PruneDeletedCustomerAudit %s: operator_actions: %w", d.id, err) + return events, notifications, err } + events += ev + notifications += nl + } + return events, notifications, nil +} + +// pruneOneDeletedCustomer is one deleted household's year-end, all or nothing: there is no day on which the log is +// gone and the id is not. +func (s *Store) pruneOneDeletedCustomer(id, cutoff string) (events, notifications int64, err error) { + tx, err := s.db.Begin() + if err != nil { + return 0, 0, err + } + defer tx.Rollback() + r, err := tx.Exec(`DELETE FROM events WHERE customer_id = ? AND created_at <= ?`, id, cutoff) + if err != nil { + return 0, 0, fmt.Errorf("PruneDeletedCustomerAudit %s: events: %w", id, err) + } + events, _ = r.RowsAffected() + r, err = tx.Exec(`DELETE FROM notification_log WHERE customer_id = ? AND created_at <= ?`, id, cutoff) + if err != nil { + return 0, 0, fmt.Errorf("PruneDeletedCustomerAudit %s: notification_log: %w", id, err) + } + notifications, _ = r.RowsAffected() + // D73 (R-1376): the operator's action log of the deleted household is an audit record of the same kind — it had + // no delete at all; it goes with the other two. + if _, err = tx.Exec(`DELETE FROM operator_actions WHERE customer_id = ? AND requested_at <= ?`, id, cutoff); err != nil { + return 0, 0, fmt.Errorf("PruneDeletedCustomerAudit %s: operator_actions: %w", id, err) + } + // D86: the journal rows of the deleted life lose their ids. legs_json first (row by row), then the id columns. + jr, err := tx.Query(`SELECT id, legs_json FROM customer_resets WHERE customer_id = ? AND started_at <= ?`, id, cutoff) + if err != nil { + return 0, 0, fmt.Errorf("PruneDeletedCustomerAudit %s: journal read: %w", id, err) + } + type leg struct { + id int64 + raw string + } + var legs []leg + for jr.Next() { + var l leg + if err := jr.Scan(&l.id, &l.raw); err != nil { + jr.Close() + return 0, 0, err + } + legs = append(legs, l) + } + jr.Close() + for _, l := range legs { + if _, err = tx.Exec(`UPDATE customer_resets SET customer_id = ?, legs_json = ? WHERE id = ?`, journalIDRemoved, scrubLegs(l.raw), l.id); err != nil { + return 0, 0, fmt.Errorf("PruneDeletedCustomerAudit %s: customer_resets: %w", id, err) + } + } + if _, err = tx.Exec(`UPDATE host_deletions SET host_id = ?, customer_id = ? WHERE customer_id = ? AND deleted_at <= ?`, + journalIDRemoved, journalIDRemoved, id, cutoff); err != nil { + return 0, 0, fmt.Errorf("PruneDeletedCustomerAudit %s: host_deletions: %w", id, err) + } + if err = tx.Commit(); err != nil { + return 0, 0, err } return events, notifications, nil } diff --git a/hub/internal/store/deleted_customer_audit_test.go b/hub/internal/store/deleted_customer_audit_test.go index e1d06c4f..8cc60d29 100644 --- a/hub/internal/store/deleted_customer_audit_test.go +++ b/hub/internal/store/deleted_customer_audit_test.go @@ -1,6 +1,7 @@ package store import ( + "strings" "testing" "time" ) @@ -63,14 +64,183 @@ func TestR901_DeletedCustomerAuditGoesAfterOneYear(t *testing.T) { } } } - // The journal row (provenance) stays. - var j int - _ = s.db.QueryRow(`SELECT COUNT(*) FROM customer_resets WHERE customer_id = 'c-old'`).Scan(&j) - if j != 1 { - t.Fatalf("the deletion journal row must stay, got %d", j) + // The journal row (provenance) stays — since D86 without its id: three rows before, three after, one with no id. + var j, noID int + _ = s.db.QueryRow(`SELECT COUNT(*) FROM customer_resets`).Scan(&j) + _ = s.db.QueryRow(`SELECT COUNT(*) FROM customer_resets WHERE customer_id = ''`).Scan(&noID) + if j != 3 || noID != 1 { + t.Fatalf("the deletion journal rows must stay (3) and c-old's must hold no id (1), got %d and %d", j, noID) } // Idempotent. if ev, nl, _ := s.PruneDeletedCustomerAudit(now, DeletedCustomerAuditKeep); ev+nl != 0 { t.Fatalf("a second run deleted %d+%d rows, want 0", ev, nl) } } + +// d86Box is a hub store with deleted and living households, for the D86 tests (`05` §14.3's test list). +type d86Box struct { + t *testing.T + s *Store + now time.Time +} + +func newD86Box(t *testing.T) *d86Box { + return &d86Box{t: t, s: newTestStore(t), now: time.Date(2027, 11, 15, 12, 0, 0, 0, time.UTC)} +} + +func (b *d86Box) ts(ago time.Duration) string { return b.now.Add(-ago).Format("2006-01-02 15:04:05") } + +func (b *d86Box) exec(q string, a ...any) { + b.t.Helper() + if _, err := b.s.db.Exec(q, a...); err != nil { + b.t.Fatal(err) + } +} + +// deleted makes a household whose delete cascade completed `ago` ago: one box delete just before, the delete journal +// row, and one audit row of each kind. +func (b *d86Box) deleted(cid, host string, ago time.Duration, legs string) { + b.exec(`INSERT INTO host_deletions (host_id, customer_id, deleted_at) VALUES (?, ?, ?)`, host, cid, b.ts(ago+time.Minute)) + b.exec(`INSERT INTO customer_resets (customer_id, started_at, completed_at, legs_json) VALUES (?, ?, ?, ?)`, cid, b.ts(ago+2*time.Minute), b.ts(ago), legs) + b.exec(`INSERT INTO events (customer_id, event_type, severity, message, created_at) VALUES (?, 'x', 'info', 'm', ?)`, cid, b.ts(ago+time.Hour)) +} + +func (b *d86Box) n(q string, a ...any) int { + b.t.Helper() + var n int + if err := b.s.db.QueryRow(q, a...).Scan(&n); err != nil { + b.t.Fatal(err) + } + return n +} + +// idAnywhere counts the journal rows that still hold the text `id` in ANY column (searched as text, so a look-alike +// id is told apart only by the caller's choice of text). +func (b *d86Box) idAnywhere(id string) int { + return b.n(`SELECT (SELECT COUNT(*) FROM customer_resets WHERE customer_id = ?1 OR instr(legs_json, ?1) > 0) + + (SELECT COUNT(*) FROM host_deletions WHERE customer_id = ?1 OR host_id = ?1 OR instr(host_id, ?1 || '-') = 1)`, id) +} + +const d86Legs = `{"hosts":"ok","residue":"ok","customer_delete":"ok"}` + +// D86, the consequence: a year after a household was deleted, no journal row holds its id or its box's id — and the +// rows are still there, with their times and step states. A look-alike id, a household deleted 364 days ago and a +// household that still exists keep everything. +func TestD86_AfterTheYearTheJournalHoldsNoIDAndKeepsItsRows(t *testing.T) { + b := newD86Box(t) + day := 24 * time.Hour + b.deleted("peti", "peti-4f2a1c", 366*day, d86Legs) + b.deleted("peti-2", "peti-2-9be001", 366*day, `{"hosts":"ok","residue":"ok","customer_delete":"failed"}`) // a look-alike whose delete never ENDED + b.deleted("anna", "anna-77aa01", 364*day, d86Legs) // one day short of the year + b.exec(`INSERT INTO host_deletions (host_id, customer_id, deleted_at) VALUES ('living-1-aa', 'living', ?)`, b.ts(500*day)) // a household that still exists, one old box delete + b.exec(`INSERT INTO customer_resets (customer_id, started_at, completed_at, legs_json) VALUES ('living', ?, ?, '{"hetzner":"ok","pbs":"ok","db_purge":"ok"}')`, b.ts(500*day), b.ts(500*day)) + rowsBefore := b.n(`SELECT (SELECT COUNT(*) FROM customer_resets) + (SELECT COUNT(*) FROM host_deletions)`) + timesBefore := b.n(`SELECT COUNT(*) FROM customer_resets WHERE completed_at = ?`, b.ts(366*day)) + + if _, _, err := b.s.PruneDeletedCustomerAudit(b.now, DeletedCustomerAuditKeep); err != nil { + t.Fatal(err) + } + if got := b.n(`SELECT COUNT(*) FROM customer_resets WHERE customer_id = 'peti'`) + b.n(`SELECT COUNT(*) FROM host_deletions WHERE customer_id = 'peti' OR host_id = 'peti-4f2a1c'`); got != 0 { + t.Fatalf("a year after the deletion %d journal row(s) still hold the household's id or its box's id", got) + } + if got := b.n(`SELECT (SELECT COUNT(*) FROM customer_resets) + (SELECT COUNT(*) FROM host_deletions)`); got != rowsBefore { + t.Fatalf("the journal lost rows: %d before, %d after — the id goes, the row stays", rowsBefore, got) + } + if got := b.n(`SELECT COUNT(*) FROM customer_resets WHERE completed_at = ?`, b.ts(366*day)); got != timesBefore { + t.Fatalf("the journal lost its times: %d rows with the completion time before, %d after", timesBefore, got) + } + if got := b.n(`SELECT COUNT(*) FROM customer_resets WHERE customer_id = '' AND legs_json = ?`, `{"customer_delete":"ok","hosts":"ok","residue":"ok"}`); got != 1 { + t.Fatalf("the step states of the deleted household's row are not kept as they were: %d", got) + } + for id, want := range map[string]int{"peti-2": 2, "anna": 2, "living": 2} { + if got := b.idAnywhere(id); got != want { + t.Errorf("%s: %d journal row(s) hold the id after the run, want %d (nothing of it may change)", id, got, want) + } + } + // the audit rows went in the same step (the log is never gone with the id still there, or the other way round) + if b.n(`SELECT COUNT(*) FROM events WHERE customer_id = 'peti'`) != 0 || b.n(`SELECT COUNT(*) FROM events WHERE customer_id = 'anna'`) != 1 { + t.Fatal("the audit rows and the journal ids did not go together") + } + // a second run changes nothing and does not read the row with no id as a household + if ev, nl, err := b.s.PruneDeletedCustomerAudit(b.now, DeletedCustomerAuditKeep); err != nil || ev+nl != 0 { + t.Fatalf("second run: %d+%d rows, err %v", ev, nl, err) + } +} + +// An id used AGAIN for a new household: the new life's journal rows are read by live code (the escrow carry, a later +// delete's purge) and must keep their ids when the OLD life's year ends. +func TestD86_AnIDUsedAgainKeepsTheJournalOfItsNewLife(t *testing.T) { + b := newD86Box(t) + day := 24 * time.Hour + b.deleted("tester4", "tester4-46836f", 400*day, d86Legs) + // the new life: the same id, a box deleted last week, a RESET last month + b.exec(`INSERT INTO host_deletions (host_id, customer_id, deleted_at, escrow_acked) VALUES ('tester4-2f0c25', 'tester4', ?, 1)`, b.ts(7*day)) + b.exec(`INSERT INTO customer_resets (customer_id, started_at, completed_at, legs_json) VALUES ('tester4', ?, ?, '{"hetzner":"ok","pbs":"ok","db_purge":"ok"}')`, b.ts(30*day), b.ts(30*day)) + if _, _, err := b.s.PruneDeletedCustomerAudit(b.now, DeletedCustomerAuditKeep); err != nil { + t.Fatal(err) + } + if got := b.n(`SELECT COUNT(*) FROM host_deletions WHERE customer_id = 'tester4' AND host_id = 'tester4-2f0c25'`); got != 1 { + t.Fatalf("the NEW life's box delete lost its id (%d) — the escrow carry of the living household reads it", got) + } + if got := b.n(`SELECT COUNT(*) FROM customer_resets WHERE customer_id = 'tester4'`); got != 1 { + t.Fatalf("the NEW life's RESET row: %d with the id, want 1", got) + } + if got := b.n(`SELECT COUNT(*) FROM host_deletions WHERE host_id = 'tester4-46836f'`) + b.n(`SELECT COUNT(*) FROM customer_resets WHERE customer_id = '' `); got != 1 { + t.Fatalf("the OLD life: want its box id gone and exactly one journal row with no id, got %d", got) + } +} + +// A household with the id that an emptied row holds cannot exist — and a household NAMED like a marker word must not be +// read as a deleted one. With a word in place of the empty string, the daily run would delete this household's log. +func TestD86_TheEmptiedRowIsNeverReadAsAHousehold(t *testing.T) { + b := newD86Box(t) + day := 24 * time.Hour + b.deleted("old", "old-aaaa01", 400*day, d86Legs) + if _, _, err := b.s.PruneDeletedCustomerAudit(b.now, DeletedCustomerAuditKeep); err != nil { + t.Fatal(err) + } + for _, word := range []string{"removed", "deleted", "gone", ""} { + b.exec(`INSERT INTO events (customer_id, event_type, severity, message, created_at) VALUES (?, 'x', 'info', 'm', ?)`, word, b.ts(500*day)) + } + if ev, _, err := b.s.PruneDeletedCustomerAudit(b.now.Add(day), DeletedCustomerAuditKeep); err != nil || ev != 0 { + t.Fatalf("the next day's run deleted %d event row(s) of a household it took the emptied journal row for (err %v)", ev, err) + } + if got := b.n(`SELECT COUNT(*) FROM customer_resets WHERE customer_id IN ('removed', 'deleted', 'gone')`); got != 0 { + t.Fatalf("the journal holds a WORD in place of the id (%d row(s)) — a household can be given that word as its id", got) + } +} + +// The step field holds state words only. If a later writer ever puts a name there, the year-end still leaves none. +func TestD86_TheStepFieldKeepsOnlyPlainWords(t *testing.T) { + b := newD86Box(t) + b.deleted("kata", "kata-123456", 400*24*time.Hour, `{"hosts":"ok","hetzner":"sub-account kata.kovacs@example.com removed","customer_delete":"ok","Kata Kovacs":"ok"}`) + if _, _, err := b.s.PruneDeletedCustomerAudit(b.now, DeletedCustomerAuditKeep); err != nil { + t.Fatal(err) + } + var legs string + if err := b.s.db.QueryRow(`SELECT legs_json FROM customer_resets WHERE customer_id = ''`).Scan(&legs); err != nil { + t.Fatal(err) + } + if legs != `{"customer_delete":"ok","hetzner":"other","hosts":"ok","other":"ok"}` { + t.Fatalf("legs_json after the year = %s", legs) + } +} + +// All or nothing: when the journal step fails, the audit rows of that household are still there (no day on which the +// log is gone and the id is not). +func TestD86_TheLogAndTheIDGoInOneTransaction(t *testing.T) { + b := newD86Box(t) + b.deleted("mari", "mari-abcdef", 400*24*time.Hour, d86Legs) + b.exec(`CREATE TRIGGER d86_fail BEFORE UPDATE ON host_deletions BEGIN SELECT RAISE(ABORT, 'staged failure'); END`) + // the error must be the STAGED one from the journal step — any other error (a lock) would mean the step was never + // reached, and the test would prove nothing + if _, _, err := b.s.PruneDeletedCustomerAudit(b.now, DeletedCustomerAuditKeep); err == nil || !strings.Contains(err.Error(), "staged failure") { + t.Fatalf("the staged failure of the journal step was not returned: %v", err) + } + if got := b.n(`SELECT COUNT(*) FROM events WHERE customer_id = 'mari'`); got != 1 { + t.Fatalf("the audit rows went (%d left) although the journal step failed", got) + } + if got := b.n(`SELECT COUNT(*) FROM customer_resets WHERE customer_id = 'mari'`); got != 1 { + t.Fatalf("the journal row lost its id although the step failed: %d", got) + } +}