diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index 0cc00818..a27bad90 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -402,6 +402,41 @@ attempts, 7 s apart, a log line each) in the provision and the F-14 re-issue pat the operator-only `pbsdr_provision_failed` („… save the customer config to retry"). Live 2026-10-26 on hub rc.1: a fresh Tester4 install provisioned the tier on the first try. +### 14.1 The off-site account when its tier is switched off [FACT — built, not released: hub `main` after v0.181.0, 2026-11-14; D71, `09` §3 decision 291, R-1375] + +The first off-site tier (the app copies) is a sub-account at the storage provider. The household's config form +rebuilds the whole config at every save, so until this change an UNTICK dropped the `offsite` block and the hub forgot +the account: a later RESET or delete skipped it, and the account stayed at the provider — paid for, with the +household's copies in it, and no record anywhere (endings EH06 and EH01). + +- **An untick takes nothing down.** No provider call, no purge. This is the wrong case the tests pin first + (`TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown`). +- **The hub keeps the record.** `offsite.KeepAccountRecord` runs at every save: a tier that was ticked AND provisioned + and is now unticked leaves `offsite_account` in the config (type, user, host, the time of the untick); later saves + carry it. It is not an `offsite` block, so the box is handed the tier as off. A tier that was ticked and never + provisioned made no account and leaves no record. +- **RESET and delete take the account down when a record exists** — ticked or not (`web.offsiteChoice` reads the + tick OR the record; both previews name the account). The folder is purged first, then the account is deleted, as + before (R-32); the record goes with the account. +- **Ticking again** takes the same account back into use (the provider is asked by the household's label) and the + record is gone: the descriptor is the record again. +- **The config form says it:** „The off-site tier is OFF, and this household's account still exists at the provider”. + +**An account that lost its record before this release** (unticked, then saved, before the fix): the hub cannot know +it. One act finds them — at the provider's page, read each sub-account's label `felhom-customer=`; for each one +whose household has the tier unticked in the hub, tick the tier again and save (the same account comes back, and its +record with it), then untick if it shall stay off. For a household that is gone: remove the account by hand, by its +label (`audits/RETIRE-peti-2026-09-25.md` A3). `runbooks/household-leaves.md`, „The untick trap”. + +**The second off-site tier** (the whole-box copies on ep0) never had this fault: switching it off keeps its place and +its descriptor (`enabled:false`, coordinates kept — `TestPBSDR_DisableKeepsTenancy`), and RESET and delete take it +down by the customer id, ticked or not. + +**Not proven:** on a deployed hub, or against the real provider. The tests run the real save, the real RESET and the +repository's fake provider (`hetznerapi.Fake`); 9 mutations seen red, one of them only after a missing test case was +added (`audits/night-build-2026-11-14/B/redproof-hub-offsite-record.txt`). + + ## 15. Customer e-mails — what the hub writes to a household [hub v0.118.0, R-558] **The section this document did not have.** The hub composes every sentence a household reads before diff --git a/documentation/audits/night-build-2026-11-14/B/redproof-hub-offsite-record.txt b/documentation/audits/night-build-2026-11-14/B/redproof-hub-offsite-record.txt new file mode 100644 index 00000000..fed8f73d --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/B/redproof-hub-offsite-record.txt @@ -0,0 +1,44 @@ +== MUTATION: THE WRONG CASE: an untick takes the account down + RED on the mutant, GREEN on the real file + --- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.05s) + d71_offsite_account_record_test.go:81: AN UNTICK TOOK SOMETHING DOWN: 0 account(s) left (want 1), 1 purge call(s) (want 0) +== MUTATION: the record is not kept at the save (the old behaviour: the hub forgets the account) + RED on the mutant, GREEN on the real file + --- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.05s) + d71_offsite_account_record_test.go:89: the account's record did not survive the untick and two saves: in {"logging":{"level":"debug"}} + +2 mutations, 0 not a red-proof +== MUTATION: RESET reads only the tick (an unticked account is skipped) + RED on the mutant, GREEN on the real file + --- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.05s) + d71_offsite_account_record_test.go:107: the reset preview does not name the kept account u629193-sub1000: {"claim_present":false,"customer_id":"acme","dr_recipe_present":false,"escrow_ack_required":false,"host_count":0,"offsite_en + d71_offsite_account_record_test.go:107: the delete preview does not name the kept account u629193-sub1000: {"claim_present":false,"cloudflare_manual":["the Cloudflare tunnel that serves acme.hu","the DNS records of acme.hu (the ap + d71_offsite_account_record_test.go:123: after the RESET: 1 account(s) left (want 0), 0 purge(s) (want 1) — the unticked account was skipped + +1 mutations, 0 not a red-proof +== MUTATION: the record is written once and lost at the next save + RED on the mutant, GREEN on the real file + --- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.04s) + d71_offsite_account_record_test.go:89: the account's record did not survive the untick and two saves: in {"logging":{"level":"debug"}} +== MUTATION: a ticked tier keeps the old untick record + *** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0) +== MUTATION: a tier that was never provisioned gets a record + RED on the mutant, GREEN on the real file + --- FAIL: TestKeepAccountRecord_FourCases (0.00s) + account_record_test.go:38: ticked, never provisioned, then unticked: no account was made, no record: a record &{Type:shared User: Host: UntickedAt:2026-11-14T20:00:00Z}, want none ({"git":{"username":"x"},"offsite_account":{"type" +== MUTATION: the record outlives the account after a RESET + RED on the mutant, GREEN on the real file + --- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.05s) + d71_offsite_account_record_test.go:131: the record outlived the account: {"logging":{"level":"debug"},"offsite_account":{"type":"shared","user":"u629193-sub1000","host":"u629193-sub1000.your-storagebox.de","unticked_at":"2026-11-1 +== MUTATION: the kept record is written as an offsite block (the box would read it) + RED on the mutant, GREEN on the real file + --- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.05s) + d71_offsite_account_record_test.go:89: the account's record did not survive the untick and two saves: in {"logging":{"level":"debug"}} + +5 mutations, 1 not a red-proof +== MUTATION: a ticked tier keeps the old untick record (done again: the first run of this mutation was GREEN — the table had no case for it; the case was added) + RED on the mutant, GREEN on the real file + --- FAIL: TestKeepAccountRecord_FourCases (0.00s) + account_record_test.go:40: a config that carries BOTH a ticked tier and a record: the record goes (the descriptor is the record): a record &{Type:shared User:u1-sub3 Host: UntickedAt:}, want none ({"offsite":{"enabled":true,"type" + +1 mutations, 0 not a red-proof diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index f6c204ae..7d6a2766 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -354,7 +354,7 @@ stopping line that lies. | **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CH23-6). | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC | | **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC | | **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN **2026-11-02 read (`audits/security-2026-11-02/part4/hub-slow-lines-2026-11-01-02.txt`; the log covers only the 31 h since DooPlex restarted on 2026-11-01 16:11 CET, the lines from 10-30 are gone with the old pod):** 35 lines, 15 statements. What repeats: `VACUUM INTO` 5× (avg 110 s, the 6-hourly snapshot, expected); `GetHostReportsSince` 9× (avg 7.9 s, max 26.6 s); `GetCustomer`'s newest report 3× (1.1 s). Once, at the cold start: the prune of `events` (10.4 s) and `app_log_issues` (21 s, again 04:30), the OS-release and host lists (2–22 s). No fix before the full week (11-06). **2026-11-10 read (the hub's start after a deploy, `audits/release-space-fixes-2026-11-10/hub/10-watch.txt`):** 11 slow lines in the first 2 minutes of the new pod, the longest 19.1 s (`GetHostReportsSince`), then the OS-report reads (3–4 s) and the two prunes (2.7 s, 4.2 s); none in the 28 minutes after. The 13:02 UTC snapshot's `VACUUM INTO` took 127.8 s. The same statements as the 11-02 read; no new one. | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC | -| **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC | +| **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | NARROWED — part 1 (D71) is built, not released. **2026-11-14 (night build):** the hub keeps the record of an off-site account after the tier is unticked; RESET and delete take the account down when a record exists; an untick alone takes nothing down (pinned by a test); the config form says that the account still exists (`05` §14.1). The runbook „a household or a tester leaves” is written (D74, `runbooks/household-leaves.md`). **For the operator, once, after the release:** at the provider's page read each sub-account's label `felhom-customer=`; for a household whose tier is unticked in the hub, tick it again and save — the account's record comes back. LEFT: part (2) has no fault (the second tier keeps its record, read in source and its test); part (3) is the runbook; a household mail at leaving stays R-810. OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC | | **R-1376** | Hub & operator | P3 | **Records of a thing that is gone stay in six places.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH00, EH02, EH21, EA00, EA11, EA27; read in source, `documentation/audits/endings-list-2026-11-10/evidence/hub-delete-statements.txt`; no hub database was read). (1) **Hub:** thirteen tables have no delete statement anywhere. Deleting a box leaves its update settings, its update reports and its kernel notices; deleting a household leaves its off-site key rows, off-site window rows and abandon requests. Most are small; the update reports of each box grow with time. (2) **Host uninstall** leaves the mount units and host mounts of household drives (said when R-179 closed; no open row carried it). (3) **A drive wiped by a signed job** keeps a record and a mount unit that name the old filesystem (agent `internal/signedjobs/wipe.go:61-110`). (4) **The token of a destroyed guest** stays valid until a new guest takes its number: the token store has no remove (agent `internal/localapi/tokenstore.go:127-210`). **Members (answers marked N):** EH00-4, EH02-4, EH21-4, EA00-4, EA11-4, EA27-4. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D73 = yes, with one exception; `09` §3 decision 293).** Not built. | — | Hub (D73): add the tables with no delete to the two purges and to the daily clean-up; the delete journal (`host_deletions`, `customer_resets`) stays on purpose, and the privacy text says so. Agent: remove a guest's token when the guest is destroyed. Parts (2) and (3) have no ruling of their own. | CC | | **R-1386** | Hub & operator | P3 | **Timed jobs with no time limit and no measured duration (25 answers).** Found 2026-11-11 (the Clock list, class W4, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source; the hub was not measured, by ruling). The hub's 60-second sweep runs its checks one after the other with no limit for each: one slow check delays all the others, the liveness check too (CH19-3, CH19-8, CH01-3, CH08-3, CH08-8, CH10-3, CH13-8, CH18-3, CH06-8, CH02-3); the 07:10 audit has 10 minutes, measured with three boxes only (CH11-3, CH11-8). On the box: the interval jobs have no limit of their own (CC11-3, CC12-3, CC13-3, CC14-3, CC15-3, CC16-3, CC20-3, CA06-3, CA10-3, CA10-8, CA13-8, CA15-3, CA19-3). Every measured duration is from a small test store. | OPEN — filed 2026-11-11 (clock list) | — | Hub: log each sweep step's duration and alarm when one sweep takes longer than its interval; then read a week of numbers and set limits. Box: the same for the controller's interval jobs. | CC | | **R-1393** | Hub & operator | P4 | **For about 8 minutes after a hub restart its database statements take 1 to 32 seconds; the System page took 12.7 s and a box's enrolment 11.4 s.** Measured 2026-11-14 at the 0.181.0 deploy (`audits/release-1-dates-2026-11-14/hub/05-watch.txt`): 24 `slow statement` lines from 09:58 to 10:06 UTC, none in the 20 minutes after. The database file is 678 MB; the slow ones read `host_reports`, `os_reports` and `events` (whole report JSON per row). One 32-second statement ran while CC copied the database out of the pod for a read (CC's own load). No report was lost that CC could see; not measured: whether a box's report timed out in that window. | OPEN — filed 2026-11-14 | — | measure a restart with no copy running; then decide: an index or a shorter keep on `host_reports`/`os_reports`, or accept | CC | diff --git a/documentation/runbooks/household-leaves.md b/documentation/runbooks/household-leaves.md new file mode 100644 index 00000000..8f85ed81 --- /dev/null +++ b/documentation/runbooks/household-leaves.md @@ -0,0 +1,131 @@ +# A household or a tester leaves — the runbook + +> **Status: written 2026-11-14 from source and from one real use; NOT run as a whole.** Operator ruling D74 +> (`09-update-architecture.md` §3 decision 294, R-1375). Each step says **PROVEN** (with where) or **NOT PROVEN**. +> A step that is NOT PROVEN is not a wrong step: nobody has seen it work. Read its line before you press. +> Source: the endings list (`audits/endings-list-2026-11-10/TABLE.md`, endings EH00 to EH06) and the one real +> retirement (`audits/RETIRE-peti-2026-09-25.md`). The design of what the hub removes is `05-hub-architecture.md` §14. + +## What this is, and what it is not + +A household leaves Felhom, or a tester stops testing. You remove what Felhom holds of them: in the hub, at the +off-site provider, on ep0, and at Cloudflare. **There is no single button.** The hub has five doors; three are +buttons, one is a tick box, one is by hand. + +It is NOT for a household that starts again on a new machine. For that, use RESET only (door 2) and stop. + +## Before you start — four questions + +1. **Does the household want its data?** The box holds the data. The hub holds none of it. Tell the household to + copy what it wants off the box (the file manager, or each app's own export) BEFORE you start. After door 3 the + off-site copies are gone and there is no undo. **NOT PROVEN:** no text for the household exists (capability E16; + R-810). Write to them by hand. +2. **Is the box still on?** An ONLINE box refuses doors 1 and 3. Ask the household to switch the box off, or wait + until the hub shows it as down (a silent box can be deleted after 6 minutes). PROVEN in tests + (`hub/internal/web/hosts.go`, the 409 gate; `customer_delete_test.go` GatesFailClosed). +3. **Was the off-site tier ever ticked for this household, and is it ticked now?** Read the household's config page. + If it was ticked once and is unticked now, see „The untick trap” below before door 3. +4. **Who is it?** Write the customer id and the host id on paper. You type both. + +## The five doors, in the order you use them + +| # | Door | Where | What it ends | Ending | +|---|---|---|---|---| +| 1 | Delete the box | hub → Hosts → the box → Delete | the box's record | EH00 | +| 2 | RESET the household | hub → the household → Reset | everything the box made, at the hub and outside; the household's record stays | EH01 | +| 3 | Delete the household | hub → the household → Delete customer… | doors 1 and 2 for every box, then the record itself | EH02 | +| 4 | Untick a tier | hub → the household → Edit | the tier is no longer offered to the box; NOTHING is removed | EH06 | +| 5 | Cloudflare, by hand | Cloudflare's dashboard | the tunnel, the DNS records, the API token | EH05 | + +Doors 2 and 3 open two more by themselves: the off-site account (EH03) and the copies on ep0 (EH04). + +**For a household that leaves: door 3, then door 5.** Door 3 does doors 1 and 2 for you. +**For a tester who stops: the same.** There is no other path for a tester (the endings list, „a tester ends”). + +## The steps + +### Step 1 — the household copies its data off the box. NOT PROVEN (no product path; by hand). + +### Step 2 — the box is off, or silent for 6 minutes. +PROVEN in tests: an online box refuses (`TestCustomerDelete…GatesFailClosed`). NOT PROVEN on a real leaving +household. + +### Step 3 — read the preview. hub → the household → Delete customer… +The dialog shows what will go: the boxes, the off-site account, the residue count, and the Cloudflare objects you +must remove by hand. **Read the off-site line.** It must name an account when the household ever had the off-site +tier. If it says „none” and you know there was one: STOP, go to „The untick trap”. +PROVEN: the preview was read at the one real use (`RETIRE-peti-2026-09-25.md` A3 step 3: 0 hosts, the sub-account, +residue 1,519). The Cloudflare line: PROVEN in a test only (`TestR688_PreviewNamesCloudflareByHand`). + +### Step 4 — tick the three boxes, type the customer id, press. +The hub then runs, in this order: the journal entry → every box deleted (door 1) → the off-site account taken down +→ the copies on ep0 taken down → the claim reset → the hub's rows purged → the record and ALL key packages purged. +- Door 1 for each box: PROVEN in tests (`TestDeleteHost_CascadeAndIsolation`); a real box delete is routine + (Tester4, 2026-11-11 and 11-14, by the operator). +- The off-site account: **PROVEN once on a real account** (`RETIRE-peti-2026-09-25.md`: sub-account 269130 removed, + the pool's other accounts and their sizes unchanged). That account held no backup. **NOT PROVEN: an account that + holds a real backup.** The hub empties the folder through the account's own login first, then deletes the account; + if the emptying fails the account is KEPT and the step fails (R-32) — then press again. +- The copies on ep0: **NOT PROVEN on real copies** (at the one real use there was nothing on ep0: `existed=false`). + Tests only (`customer_delete_test.go`). The second copy of ep0 on DooPlex is NOT removed by this and was not + measured (`runbooks/ep0-datastore-copy.md`). +- The purge of the hub's rows: PROVEN once (482 reports, 1,036 telemetry rows, 1 log tail gone; journal #20). + **Known gap, R-1376:** rows in 13 tables stay (the box's update settings and reports, its kernel notices, the + off-site key and window rows, the abandon requests). Ruled to be removed (D73, `09` §3 decision 293); read the + row for whether that is released — until then they stay. + +### Step 5 — if a leg fails: press again. +The dialog offers Resume. A leg that is done is not done again. Nothing is half forgotten: the journal names the leg. +PROVEN in tests (`…ResumesAfterExternalFailure`). NOT PROVEN live (`_recovery-inventory-2026-07-28.md`: „the +end-to-end live leg is not run”). + +### Step 6 — check the provider by hand. 5 minutes. +At the off-site provider's page: the household's sub-account is gone; the others are there; the pool's size did not +jump. PROVEN as a check (it is what `RETIRE-peti-2026-09-25.md` A4 did). **Always do it when the journal says +`skipped_no_config`** — the hub could not know the account then. + +### Step 7 — door 5, Cloudflare, by hand. NOT PROVEN (never done; no code does it; the operator accepted that, R-688). +From the dialog's list: 1. delete the tunnel; 2. delete the DNS records that point at it; 3. delete the household's +API token; 4. if the zone was added only for this household, remove the zone or leave it to the household (it is +THEIR domain — ask them). If you do nothing: the names point at a tunnel that no longer runs; a visitor sees +Cloudflare's error page. No data is exposed. + +### Step 8 — what stays, on purpose. Do not „clean” these. +- The audit trail in the hub: the delete and reset journals, the host-deletion records, the events and the + notification log. The last two go by themselves after one year (R-901, PROVEN in a test). +- The hub's own database snapshots: they hold the household's rows for up to 8 weeks on ep0. That is the only way + back, and it is a retention fact you must be able to state to the household. +- The household's box and its drives: theirs. Felhom removes nothing on the box. If the household wants the box + wiped, that is a new install by them, or the box's own „factory” path — NOT PROVEN as part of leaving. + +### Step 9 — a tester only: what to collect. NOT PROVEN (no list existed; this is the first). +Before step 4, ask the tester for: what did not work (in their words); whether they want to come back later (then +use RESET, door 2, and keep the record); the box's hub page saved as a file if a finding is open in the register. +After step 7: remove the tester from `runbooks/target-selection.md` (as Peti's entry was), and from `STATUS.md` if +named there. + +## The untick trap (door 4) — read this if the off-site tier was unticked + +Unticking the off-site tier removes nothing at the provider. **Until the fix is released, it also made the hub +forget the account:** a later RESET or delete then skipped it, and the account stayed, paid for, with the +household's copies in it, and no record in the hub (R-1375, endings EH06 and EH01). + +- **Fix built, not released (hub `main`, 2026-11-14, D71):** the hub keeps the account's record after an untick, and + RESET and delete take the account down when a record exists. +- **Until that release, and for any household unticked before it:** before door 3, tick the off-site tier again and + save (this is ending EH06's own undo and gives the record back), THEN do door 3. NOT PROVEN on a real household. +- **If the record is already lost** (unticked, then RESET or deleted, before the fix): the account has the label + `felhom-customer=` at the provider. Find it by the label, never by its number, and remove it by hand + as `RETIRE-peti-2026-09-25.md` A3 did. PROVEN once (that record). + +## The second off-site tier (the whole-box copies on ep0) + +Switching it off keeps the household's place and copies on ep0, on purpose (PROVEN in a test, +`TestPBSDR_DisableKeepsTenancy`). The hub finds them by the customer id, so door 2 and door 3 take them down whether +the tier is ticked or not. Nothing to do by hand here. + +## What is not in this runbook, and why + +- A mail to the household that says what was removed. It does not exist (R-810). Write by hand. +- A refund or an invoice. Money is the operator's. +- A wipe of the household's box. Not Felhom's to do. diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 9e345543..5a429c81 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -3,6 +3,15 @@ - `bundlefiles`: the line „bundle X holds the same files as the vouched Y” is logged once per pair (INFO), not at every view of the System page (it was 6 DEBUG lines per view, seen at the 0.181.0 deploy). No behaviour change. - **Release 3 („the hub tells you”), built 2026-11-14 night, NOT deployed. Every new alarm is operator-only.** +- **The record of an off-site account outlives an untick (R-1375 part 1, D71, `09` §3 decision 291; `05` §14.1).** + The config form rebuilds the whole config at a save, so unticking the off-site tier dropped the `offsite` block and + the hub forgot the account; a later RESET or delete skipped it. Now `offsite.KeepAccountRecord` leaves + `offsite_account` (type, user, host, time) in the config and carries it through later saves; `web.offsiteChoice` + reads the tick OR the record, so RESET, delete and both previews find the account; the RESET's descriptor leg drops + the record with the account; the config form says that the account still exists. **An untick alone takes nothing + down** — pinned first. Tests `TestD71_*` (the real save and RESET over `hetznerapi.Fake`), + `TestKeepAccountRecord_FourCases`; 9 mutations seen red (one stayed green at first: the table had no case for it; + the case was added and the mutation done again). - **Four silent failures tell the operator (R-1383, D78, `09` §3 decision 299; `05` §16.6).** No new table. (1) `hub_daily_job_skipped`: `monitor.DailyLedger` keeps across a restart when `deadline-check` and `offsite-key-audit` last ran; a slot that passed while the hub was down is told once at the next start (the check diff --git a/hub/internal/offsite/account_record_test.go b/hub/internal/offsite/account_record_test.go new file mode 100644 index 00000000..b78133b6 --- /dev/null +++ b/hub/internal/offsite/account_record_test.go @@ -0,0 +1,51 @@ +package offsite + +import ( + "strings" + "testing" + "time" +) + +// D71 (R-1375): the four cases of KeepAccountRecord, as a table. The web tests (TestD71_*) prove the consequence +// through the real save and the real RESET; this pins the rule itself. +func TestKeepAccountRecord_FourCases(t *testing.T) { + now := time.Date(2026, 11, 14, 20, 0, 0, 0, time.UTC) + const ( + provisioned = `{"git":{"username":"x"},"offsite":{"enabled":true,"type":"shared","host":"u1-sub3.example","user":"u1-sub3","port":23}}` + choiceOnly = `{"offsite":{"enabled":true,"type":"shared","quota_gb":100}}` // ticked, never provisioned (or after a RESET) + withRecord = `{"offsite_account":{"type":"shared","user":"u1-sub3","host":"u1-sub3.example","unticked_at":"2026-11-01T00:00:00Z"}}` + unticked = `{"git":{"username":"x"}}` + ) + for _, c := range []struct { + name, prev, next string + wantUser, wantAt string // "" = no record + keepsOther string + }{ + {"ticked and provisioned, then unticked: a record from the old descriptor", provisioned, unticked, "u1-sub3", "2026-11-14T20:00:00Z", `"git"`}, + {"a record, still unticked: carried unchanged (the form does not know it)", withRecord, unticked, "u1-sub3", "2026-11-01T00:00:00Z", `"git"`}, + {"ticked again: no record", withRecord, provisioned, "", "", `"offsite"`}, + {"a config that carries BOTH a ticked tier and a record: the record goes (the descriptor is the record)", withRecord, + `{"offsite":{"enabled":true,"type":"shared","user":"u1-sub3"},"offsite_account":{"type":"shared","user":"u1-sub3"}}`, "", "", `"offsite"`}, + {"ticked, never provisioned, then unticked: no account was made, no record", choiceOnly, unticked, "", "", `"git"`}, + {"never ticked: nothing", `{}`, unticked, "", "", `"git"`}, + {"an empty previous config", ``, `{}`, "", "", ``}, + } { + out, err := KeepAccountRecord(c.prev, c.next, now) + if err != nil { + t.Fatalf("%s: %v", c.name, err) + } + rec := ReadAccountRecord(out) + switch { + case c.wantUser == "" && rec != nil: + t.Errorf("%s: a record %+v, want none (%s)", c.name, rec, out) + case c.wantUser != "" && (rec == nil || rec.User != c.wantUser || rec.UntickedAt != c.wantAt || rec.Type != "shared"): + t.Errorf("%s: record %+v, want user %s unticked_at %s (%s)", c.name, rec, c.wantUser, c.wantAt, out) + } + if c.keepsOther != "" && !strings.Contains(out, c.keepsOther) { + t.Errorf("%s: the other keys were lost: %s", c.name, out) + } + } + if _, err := KeepAccountRecord(provisioned, `{not json`, now); err == nil { + t.Error("a config that cannot be read must be an error (the save is then refused), not a silent loss of the record") + } +} diff --git a/hub/internal/offsite/offsite.go b/hub/internal/offsite/offsite.go index c6170e7c..1dbe37e0 100644 --- a/hub/internal/offsite/offsite.go +++ b/hub/internal/offsite/offsite.go @@ -514,6 +514,8 @@ func ClearProvisionedDescriptor(configJSON string) (string, error) { if err != nil { return "", err } + // D71: the record of an unticked tier's account dies with the account too (RESET took it down before this leg). + configJSON = dropAccountRecord(configJSON) if cur == nil { return configJSON, nil // no offsite block — nothing to clear } @@ -521,6 +523,97 @@ func ClearProvisionedDescriptor(configJSON string) (string, error) { return MergeDescriptor(configJSON, cleared) } +func dropAccountRecord(configJSON string) string { + obj := map[string]json.RawMessage{} + if json.Unmarshal([]byte(configJSON), &obj) != nil { + return configJSON + } + if _, ok := obj[AccountRecordKey]; !ok { + return configJSON + } + delete(obj, AccountRecordKey) + out, err := json.Marshal(obj) + if err != nil { + return configJSON + } + return string(out) +} + +// AccountRecordKey is the ConfigJSON key that holds the record of an off-site account whose tier was UNTICKED +// (D71, `09` §3 decision 291, R-1375; `06-offsite-connectivity.md`). The box does not read it. +const AccountRecordKey = "offsite_account" + +// AccountRecord says: an off-site account for this household still exists at the provider although the tier is +// switched off. Before it, unticking the tier dropped the whole `offsite` block (the form rebuilds ConfigJSON), so +// the hub forgot the account: a later RESET or delete skipped it and the account stayed, paid for, with the +// household's copies in it and no record anywhere (endings EH06-4, EH01-4). Non-secret. +type AccountRecord struct { + Type string `json:"type,omitempty"` // shared | dedicated — what Deprovision needs + User string `json:"user,omitempty"` + Host string `json:"host,omitempty"` + UntickedAt string `json:"unticked_at,omitempty"` // RFC3339 UTC +} + +// ReadAccountRecord returns the record, or nil when there is none (or it cannot be read). +func ReadAccountRecord(configJSON string) *AccountRecord { + obj := map[string]json.RawMessage{} + if json.Unmarshal([]byte(configJSON), &obj) != nil { + return nil + } + raw, ok := obj[AccountRecordKey] + if !ok { + return nil + } + var r AccountRecord + if json.Unmarshal(raw, &r) != nil { + return nil + } + return &r +} + +// KeepAccountRecord is called on every save of a household's config, with the config as it WAS and as the form +// built it. It returns the new config with the account record carried: +// - the tier is ticked in the new config: no record (the descriptor itself is the record); +// - the tier was ticked AND PROVISIONED before and is unticked now: a record is written from the old descriptor; +// - a record was there before and the tier is still unticked: the record is carried (the form does not know it); +// - else: nothing — a tier that was ticked and never provisioned made no account. +// +// It calls no provider and removes nothing: AN UNTICK ALONE NEVER TAKES AN ACCOUNT OR ITS DATA DOWN. Only RESET and +// delete do (web.commitCustomerReset), and they read this record. Pinned by TestKeepAccountRecord_* and +// web TestD71_*. +func KeepAccountRecord(prevJSON, newJSON string, now time.Time) (string, error) { + obj := map[string]json.RawMessage{} + if strings.TrimSpace(newJSON) != "" { + if err := json.Unmarshal([]byte(newJSON), &obj); err != nil { + return "", fmt.Errorf("offsite: parse config_json: %w", err) + } + } + newD, err := ReadDescriptor(newJSON) + if err != nil { + return "", err + } + if newD != nil && newD.Enabled { + delete(obj, AccountRecordKey) + } else { + var rec *AccountRecord + if prevD, _ := ReadDescriptor(prevJSON); prevD != nil && prevD.Enabled && (prevD.User != "" || prevD.Host != "") { + rec = &AccountRecord{Type: prevD.Type, User: prevD.User, Host: prevD.Host, UntickedAt: now.UTC().Format(time.RFC3339)} + } else if prevRec := ReadAccountRecord(prevJSON); prevRec != nil { + rec = prevRec + } + if rec == nil { + return newJSON, nil + } + rb, _ := json.Marshal(rec) + obj[AccountRecordKey] = rb + } + out, err := json.Marshal(obj) + if err != nil { + return "", err + } + return string(out), nil +} + // ReadDescriptor extracts the non-secret offsite Descriptor from a customer's ConfigJSON, or (nil, nil) // when there is no offsite block (never provisioned). This is the AUTHORITATIVE tier/quota source — the // pool-box Σ(quota) aggregate (v0.64.0, R-5) and the RESET clear both read through it, NEVER the report diff --git a/hub/internal/web/configs.go b/hub/internal/web/configs.go index 91db6f75..b3118f88 100644 --- a/hub/internal/web/configs.go +++ b/hub/internal/web/configs.go @@ -916,6 +916,7 @@ func (s *Server) handleConfigUpdate(w http.ResponseWriter, r *http.Request, cust } } + prevConfigJSON := cfg.ConfigJSON // D71: the form rebuilds ConfigJSON; the account's record must outlive an untick cfg.ConfigJSON = buildConfigJSON(r) if err := s.applyOffsite(r.Context(), r, cfg); errors.Is(err, offsite.ErrProvisionInProgress) { @@ -928,6 +929,18 @@ func (s *Server) handleConfigUpdate(w http.ResponseWriter, r *http.Request, cust http.Error(w, "Offsite provisioning failed: "+err.Error(), http.StatusBadGateway) return } + // D71 (R-1375): unticking the off-site tier takes NOTHING down — and the hub now keeps the record of the account, + // so a later RESET or delete finds it. Fail-closed: a config we cannot carry the record into is not saved. + if kept, kerr := offsite.KeepAccountRecord(prevConfigJSON, cfg.ConfigJSON, time.Now()); kerr != nil { + s.logger.Printf("[ERROR] offsite account record for %s: %v", customerID, kerr) + http.Error(w, "Internal error", http.StatusInternalServerError) + return + } else { + if rec := offsite.ReadAccountRecord(kept); rec != nil && offsite.ReadAccountRecord(prevConfigJSON) == nil { + s.logger.Printf("[WARN] offsite tier UNTICKED for %s: nothing was removed at the provider; the account (%s, %s) stays and its record is kept — RESET or delete takes it down", customerID, rec.Type, rec.User) + } + cfg.ConfigJSON = kept + } // PBS DR tier (fail-closed; idempotent on an already-provisioned descriptor — no re-key, // no second secret, no spurious generation bump). diff --git a/hub/internal/web/customer_reset.go b/hub/internal/web/customer_reset.go index 95fe660f..78bfd2f6 100644 --- a/hub/internal/web/customer_reset.go +++ b/hub/internal/web/customer_reset.go @@ -23,6 +23,9 @@ import ( // offsiteChoice reads the customer's offsite tier selection out of config_json (the same shape the // re-issue/freeze handlers read). enabled=false means no offsite leg to run. +// offsiteChoice answers „is there an off-site account of this household to take down, and of which type?” — the tier +// is ticked, OR it was unticked and the account's record was kept (offsite.KeepAccountRecord). Read by the RESET +// commit and by both previews. Pinned by TestD71_*. func offsiteChoice(configJSON string) (enabled bool, typ string) { var o struct { Offsite struct { @@ -31,6 +34,13 @@ func offsiteChoice(configJSON string) (enabled bool, typ string) { } `json:"offsite"` } _ = json.Unmarshal([]byte(configJSON), &o) + if !o.Offsite.Enabled { + // D71 (`09` §3 decision 291, R-1375): the tier is unticked, but the hub kept the record of the account it + // made — RESET and delete take that account down too, and their previews name it. An untick alone never does. + if rec := offsite.ReadAccountRecord(configJSON); rec != nil { + return true, rec.Type + } + } return o.Offsite.Enabled, o.Offsite.Type } diff --git a/hub/internal/web/d71_offsite_account_record_test.go b/hub/internal/web/d71_offsite_account_record_test.go new file mode 100644 index 00000000..b683762d --- /dev/null +++ b/hub/internal/web/d71_offsite_account_record_test.go @@ -0,0 +1,174 @@ +package web + +import ( + "context" + "encoding/json" + "io" + "log" + "net/http" + "net/http/httptest" + "net/url" + "strings" + "testing" + + "gitea.dooplex.hu/admin/felhom-hub/internal/hetznerapi" + "gitea.dooplex.hu/admin/felhom-hub/internal/offsite" + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// d71Server: the hub's web server over the repo's FAKE provider (hetznerapi.Fake — no provider is called). purges +// counts the calls to the shared folder's purge, the step that destroys a household's off-site copies. +func d71Server(t *testing.T) (*Server, *store.Store, *hetznerapi.Fake, *int) { + t.Helper() + s, st := newTestServer(t) + api := hetznerapi.NewFake() + purges := 0 + s.SetOffsiteProvisioner(&offsite.Provisioner{ + API: api, Store: st, Scanner: webTestScanner{}, PoolBoxID: 611714, Location: "fsn1", + Logger: log.New(io.Discard, "", 0), + PurgeShared: func(context.Context, string) error { purges++; return nil }, + }) + s.SetTenantSync(&fakeTenancy{}) + if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "acme", CustomerName: "Acme", Domain: "acme.hu", + Email: "a@example.com", RetrievalPassword: "pw", APIKey: "k", ConfigJSON: "{}"}); err != nil { + t.Fatal(err) + } + return s, st, api, &purges +} + +func d71Save(t *testing.T, s *Server, form string) { + t.Helper() + w := httptest.NewRecorder() + s.handleConfigUpdate(w, postForm("/configs/acme/edit", "customer_name=Acme&domain=acme.hu&dr_tier=on"+form), "acme") + if w.Code != http.StatusSeeOther { + t.Fatalf("save (%s): %d %s", form, w.Code, w.Body.String()) + } +} + +func d71Accounts(t *testing.T, api *hetznerapi.Fake) int { + t.Helper() + subs, err := api.ListSubaccounts(context.Background(), 611714, "") + if err != nil { + t.Fatal(err) + } + return len(subs) +} + +const d71Tick = "&offsite_enabled=on&offsite_type=shared&offsite_quota_gb=50" + +// D71 (`09` §3 decision 291, R-1375). THE WRONG CASE, pinned first: an untick alone must NEVER take an account or +// its data down. Then the fix: the hub keeps the account's record through the untick (and through later saves — the +// form rebuilds the whole config), so a later RESET finds the account and takes it down; before, RESET skipped it and +// the account stayed at the provider, paid for, with no record in the hub. +// COMPANION RED-PROOFS (observed, audits/night-build-2026-11-14/B/redproof-hub-offsite-record.txt). +func TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown(t *testing.T) { + s, st, api, purges := d71Server(t) + d71Save(t, s, d71Tick) + if n := d71Accounts(t, api); n != 1 { + t.Fatalf("setup: %d account(s) at the fake provider after the tick, want 1", n) + } + cfg, _ := st.GetCustomerConfig("acme") + d, _ := offsite.ReadDescriptor(cfg.ConfigJSON) + if d == nil || d.User == "" { + t.Fatalf("setup: no provisioned descriptor: %s", cfg.ConfigJSON) + } + + // UNTICK — and two more saves after it + d71Save(t, s, "") + d71Save(t, s, "") + d71Save(t, s, "&debug_mode=on") + if n := d71Accounts(t, api); n != 1 || *purges != 0 { + t.Fatalf("AN UNTICK TOOK SOMETHING DOWN: %d account(s) left (want 1), %d purge call(s) (want 0)", n, *purges) + } + cfg, _ = st.GetCustomerConfig("acme") + if nd, _ := offsite.ReadDescriptor(cfg.ConfigJSON); nd != nil && nd.Enabled { + t.Fatalf("the tier is still enabled after the untick: %s", cfg.ConfigJSON) + } + rec := offsite.ReadAccountRecord(cfg.ConfigJSON) + if rec == nil || rec.User != d.User || rec.Type != "shared" || rec.UntickedAt == "" { + t.Fatalf("the account's record did not survive the untick and two saves: %+v in %s", rec, cfg.ConfigJSON) + } + // the box is handed the tier as OFF: the record is not an `offsite` block + var box map[string]json.RawMessage + _ = json.Unmarshal([]byte(cfg.ConfigJSON), &box) + if _, has := box["offsite"]; has { + t.Errorf("an unticked tier still carries an `offsite` block the box would read: %s", cfg.ConfigJSON) + } + // both previews name the account + for _, path := range []string{"reset", "delete"} { + rr := httptest.NewRecorder() + req := httptest.NewRequest("GET", "/configs/acme/"+path, nil) + if path == "reset" { + s.handleCustomerResetPreview(rr, req, "acme") + } else { + s.handleCustomerDeletePreview(rr, req, "acme") + } + if !strings.Contains(rr.Body.String(), `"offsite_enabled":true`) || !strings.Contains(rr.Body.String(), d.User) { + t.Errorf("the %s preview does not name the kept account %s: %s", path, d.User, rr.Body.String()) + } + } + // the form says it + page := httptest.NewRecorder() + s.renderConfigForm(page, httptest.NewRequest("GET", "/configs/acme/edit", nil), false, cfg, nil, "") + if !strings.Contains(page.Body.String(), "data-offsite-account-kept") || !strings.Contains(page.Body.String(), d.User) { + t.Errorf("the config form does not say that the account still exists") + } + + // RESET: the account goes, its folder is purged first, and the record goes with it + rr := postReset(t, s, "acme", url.Values{"confirm_id": {"acme"}}) + if rr.Code != http.StatusSeeOther { + t.Fatalf("reset: %d %s", rr.Code, rr.Body.String()) + } + if n := d71Accounts(t, api); n != 0 || *purges != 1 { + t.Fatalf("after the RESET: %d account(s) left (want 0), %d purge(s) (want 1) — the unticked account was skipped", n, *purges) + } + cr, _ := st.LatestCustomerReset("acme") + if cr == nil || cr.Legs["hetzner"] != "ok" { + t.Fatalf("journal: %+v, want the hetzner leg ok (not skipped)", cr) + } + cfg, _ = st.GetCustomerConfig("acme") + if offsite.ReadAccountRecord(cfg.ConfigJSON) != nil { + t.Errorf("the record outlived the account: %s", cfg.ConfigJSON) + } +} + +// Ticking again takes the same account back into use (the provider is asked by the household's label, so no second +// account is made) and the record is gone: the descriptor is the record again. +func TestD71_TickAgainUsesTheSameAccount(t *testing.T) { + s, st, api, purges := d71Server(t) + d71Save(t, s, d71Tick) + cfg, _ := st.GetCustomerConfig("acme") + first, _ := offsite.ReadDescriptor(cfg.ConfigJSON) + d71Save(t, s, "") + d71Save(t, s, d71Tick) + cfg, _ = st.GetCustomerConfig("acme") + again, _ := offsite.ReadDescriptor(cfg.ConfigJSON) + if again == nil || !again.Enabled || again.User != first.User { + t.Fatalf("after ticking again: %+v, want the first account %s", again, first.User) + } + if offsite.ReadAccountRecord(cfg.ConfigJSON) != nil { + t.Errorf("a ticked tier still carries the untick record: %s", cfg.ConfigJSON) + } + if n := d71Accounts(t, api); n != 1 || *purges != 0 { + t.Fatalf("accounts=%d purges=%d, want 1 and 0", n, *purges) + } +} + +// A household that never had the tier gets no record, and a RESET then calls no provider (as before). +func TestD71_NeverTickedMakesNoRecord(t *testing.T) { + s, st, api, purges := d71Server(t) + d71Save(t, s, "") + d71Save(t, s, "") + cfg, _ := st.GetCustomerConfig("acme") + if offsite.ReadAccountRecord(cfg.ConfigJSON) != nil { + t.Fatalf("a record for an account that never existed: %s", cfg.ConfigJSON) + } + if rr := postReset(t, s, "acme", url.Values{"confirm_id": {"acme"}}); rr.Code != http.StatusSeeOther { + t.Fatalf("reset: %d", rr.Code) + } + cr, _ := st.LatestCustomerReset("acme") + if cr == nil || cr.Legs["hetzner"] != "skipped" || d71Accounts(t, api) != 0 || *purges != 0 { + t.Fatalf("journal %+v, purges %d — want the hetzner leg skipped and nothing called", cr, *purges) + } + _ = http.StatusOK +} diff --git a/hub/internal/web/templates/config_form_body.html b/hub/internal/web/templates/config_form_body.html index b918d518..82ae756e 100644 --- a/hub/internal/web/templates/config_form_body.html +++ b/hub/internal/web/templates/config_form_body.html @@ -97,13 +97,14 @@ -
+

Offsite backup

+ {{with index .Overrides "offsite_account"}}

The off-site tier is OFF, and this household's account still exists at the provider: {{index . "user"}} ({{index . "type"}}), unticked {{index . "unticked_at"}}. Unticking removed nothing. A RESET or „Delete customer…” takes the account and its copies down. Tick again to use it.

{{end}}