hub (unreleased): the record of an off-site account outlives an untick; RESET and delete take the account down (R-1375, D71); runbook: a household or a tester leaves (D74)
gates / gates (push) Successful in 2m0s

An untick alone takes nothing down (pinned first). Tests run the real save and RESET over the repository's fake
provider. Not deployed.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-14 17:56:29 +01:00
1 parent 186f3fca58
commit e5c643d73d
11 files changed
+563 -2

No files matched your search

@@ -402,6 +402,41 @@ attempts, 7 s apart, a log line each) in the provision and the F-14 re-issue pat
the operator-only `pbsdr_provision_failed` („… save the customer config to retry"). Live 2026-10-26 on hub rc.1: a fresh
Tester4 install provisioned the tier on the first try.
### 14.1 The off-site account when its tier is switched off [FACT — built, not released: hub `main` after v0.181.0, 2026-11-14; D71, `09` §3 decision 291, R-1375]
The first off-site tier (the app copies) is a sub-account at the storage provider. The household's config form
rebuilds the whole config at every save, so until this change an UNTICK dropped the `offsite` block and the hub forgot
the account: a later RESET or delete skipped it, and the account stayed at the provider — paid for, with the
household's copies in it, and no record anywhere (endings EH06 and EH01).
- **An untick takes nothing down.** No provider call, no purge. This is the wrong case the tests pin first
(`TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown`).
- **The hub keeps the record.** `offsite.KeepAccountRecord` runs at every save: a tier that was ticked AND provisioned
and is now unticked leaves `offsite_account` in the config (type, user, host, the time of the untick); later saves
carry it. It is not an `offsite` block, so the box is handed the tier as off. A tier that was ticked and never
provisioned made no account and leaves no record.
- **RESET and delete take the account down when a record exists** — ticked or not (`web.offsiteChoice` reads the
tick OR the record; both previews name the account). The folder is purged first, then the account is deleted, as
before (R-32); the record goes with the account.
- **Ticking again** takes the same account back into use (the provider is asked by the household's label) and the
record is gone: the descriptor is the record again.
- **The config form says it:** „The off-site tier is OFF, and this household's account still exists at the provider”.
**An account that lost its record before this release** (unticked, then saved, before the fix): the hub cannot know
it. One act finds them — at the provider's page, read each sub-account's label `felhom-customer=<id>`; for each one
whose household has the tier unticked in the hub, tick the tier again and save (the same account comes back, and its
record with it), then untick if it shall stay off. For a household that is gone: remove the account by hand, by its
label (`audits/RETIRE-peti-2026-09-25.md` A3). `runbooks/household-leaves.md`, „The untick trap”.
**The second off-site tier** (the whole-box copies on ep0) never had this fault: switching it off keeps its place and
its descriptor (`enabled:false`, coordinates kept — `TestPBSDR_DisableKeepsTenancy`), and RESET and delete take it
down by the customer id, ticked or not.
**Not proven:** on a deployed hub, or against the real provider. The tests run the real save, the real RESET and the
repository's fake provider (`hetznerapi.Fake`); 9 mutations seen red, one of them only after a missing test case was
added (`audits/night-build-2026-11-14/B/redproof-hub-offsite-record.txt`).
## 15. Customer e-mails — what the hub writes to a household [hub v0.118.0, R-558]
**The section this document did not have.** The hub composes every sentence a household reads before
@@ -0,0 +1,44 @@
== MUTATION: THE WRONG CASE: an untick takes the account down
RED on the mutant, GREEN on the real file
--- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.05s)
d71_offsite_account_record_test.go:81: AN UNTICK TOOK SOMETHING DOWN: 0 account(s) left (want 1), 1 purge call(s) (want 0)
== MUTATION: the record is not kept at the save (the old behaviour: the hub forgets the account)
RED on the mutant, GREEN on the real file
--- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.05s)
d71_offsite_account_record_test.go:89: the account's record did not survive the untick and two saves: <nil> in {"logging":{"level":"debug"}}
2 mutations, 0 not a red-proof
== MUTATION: RESET reads only the tick (an unticked account is skipped)
RED on the mutant, GREEN on the real file
--- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.05s)
d71_offsite_account_record_test.go:107: the reset preview does not name the kept account u629193-sub1000: {"claim_present":false,"customer_id":"acme","dr_recipe_present":false,"escrow_ack_required":false,"host_count":0,"offsite_en
d71_offsite_account_record_test.go:107: the delete preview does not name the kept account u629193-sub1000: {"claim_present":false,"cloudflare_manual":["the Cloudflare tunnel that serves acme.hu","the DNS records of acme.hu (the ap
d71_offsite_account_record_test.go:123: after the RESET: 1 account(s) left (want 0), 0 purge(s) (want 1) — the unticked account was skipped
1 mutations, 0 not a red-proof
== MUTATION: the record is written once and lost at the next save
RED on the mutant, GREEN on the real file
--- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.04s)
d71_offsite_account_record_test.go:89: the account's record did not survive the untick and two saves: <nil> in {"logging":{"level":"debug"}}
== MUTATION: a ticked tier keeps the old untick record
*** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0)
== MUTATION: a tier that was never provisioned gets a record
RED on the mutant, GREEN on the real file
--- FAIL: TestKeepAccountRecord_FourCases (0.00s)
account_record_test.go:38: ticked, never provisioned, then unticked: no account was made, no record: a record &{Type:shared User: Host: UntickedAt:2026-11-14T20:00:00Z}, want none ({"git":{"username":"x"},"offsite_account":{"type"
== MUTATION: the record outlives the account after a RESET
RED on the mutant, GREEN on the real file
--- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.05s)
d71_offsite_account_record_test.go:131: the record outlived the account: {"logging":{"level":"debug"},"offsite_account":{"type":"shared","user":"u629193-sub1000","host":"u629193-sub1000.your-storagebox.de","unticked_at":"2026-11-1
== MUTATION: the kept record is written as an offsite block (the box would read it)
RED on the mutant, GREEN on the real file
--- FAIL: TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown (0.05s)
d71_offsite_account_record_test.go:89: the account's record did not survive the untick and two saves: <nil> in {"logging":{"level":"debug"}}
5 mutations, 1 not a red-proof
== MUTATION: a ticked tier keeps the old untick record (done again: the first run of this mutation was GREEN — the table had no case for it; the case was added)
RED on the mutant, GREEN on the real file
--- FAIL: TestKeepAccountRecord_FourCases (0.00s)
account_record_test.go:40: a config that carries BOTH a ticked tier and a record: the record goes (the descriptor is the record): a record &{Type:shared User:u1-sub3 Host: UntickedAt:}, want none ({"offsite":{"enabled":true,"type"
1 mutations, 0 not a red-proof
+1 -1
View File
@@ -354,7 +354,7 @@ stopping line that lies.
| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CH23-6). | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
| **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC |
| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN **2026-11-02 read (`audits/security-2026-11-02/part4/hub-slow-lines-2026-11-01-02.txt`; the log covers only the 31 h since DooPlex restarted on 2026-11-01 16:11 CET, the lines from 10-30 are gone with the old pod):** 35 lines, 15 statements. What repeats: `VACUUM INTO` 5× (avg 110 s, the 6-hourly snapshot, expected); `GetHostReportsSince` 9× (avg 7.9 s, max 26.6 s); `GetCustomer`'s newest report 3× (1.1 s). Once, at the cold start: the prune of `events` (10.4 s) and `app_log_issues` (21 s, again 04:30), the OS-release and host lists (2–22 s). No fix before the full week (11-06). **2026-11-10 read (the hub's start after a deploy, `audits/release-space-fixes-2026-11-10/hub/10-watch.txt`):** 11 slow lines in the first 2 minutes of the new pod, the longest 19.1 s (`GetHostReportsSince`), then the OS-report reads (3–4 s) and the two prunes (2.7 s, 4.2 s); none in the 28 minutes after. The 13:02 UTC snapshot's `VACUUM INTO` took 127.8 s. The same statements as the 11-02 read; no new one. | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC |
| **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC |
| **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | NARROWED — part 1 (D71) is built, not released. **2026-11-14 (night build):** the hub keeps the record of an off-site account after the tier is unticked; RESET and delete take the account down when a record exists; an untick alone takes nothing down (pinned by a test); the config form says that the account still exists (`05` §14.1). The runbook „a household or a tester leaves” is written (D74, `runbooks/household-leaves.md`). **For the operator, once, after the release:** at the provider's page read each sub-account's label `felhom-customer=<id>`; for a household whose tier is unticked in the hub, tick it again and save — the account's record comes back. LEFT: part (2) has no fault (the second tier keeps its record, read in source and its test); part (3) is the runbook; a household mail at leaving stays R-810. OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC |
| **R-1376** | Hub & operator | P3 | **Records of a thing that is gone stay in six places.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH00, EH02, EH21, EA00, EA11, EA27; read in source, `documentation/audits/endings-list-2026-11-10/evidence/hub-delete-statements.txt`; no hub database was read). (1) **Hub:** thirteen tables have no delete statement anywhere. Deleting a box leaves its update settings, its update reports and its kernel notices; deleting a household leaves its off-site key rows, off-site window rows and abandon requests. Most are small; the update reports of each box grow with time. (2) **Host uninstall** leaves the mount units and host mounts of household drives (said when R-179 closed; no open row carried it). (3) **A drive wiped by a signed job** keeps a record and a mount unit that name the old filesystem (agent `internal/signedjobs/wipe.go:61-110`). (4) **The token of a destroyed guest** stays valid until a new guest takes its number: the token store has no remove (agent `internal/localapi/tokenstore.go:127-210`). **Members (answers marked N):** EH00-4, EH02-4, EH21-4, EA00-4, EA11-4, EA27-4. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D73 = yes, with one exception; `09` §3 decision 293).** Not built. | — | Hub (D73): add the tables with no delete to the two purges and to the daily clean-up; the delete journal (`host_deletions`, `customer_resets`) stays on purpose, and the privacy text says so. Agent: remove a guest's token when the guest is destroyed. Parts (2) and (3) have no ruling of their own. | CC |
| **R-1386** | Hub & operator | P3 | **Timed jobs with no time limit and no measured duration (25 answers).** Found 2026-11-11 (the Clock list, class W4, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source; the hub was not measured, by ruling). The hub's 60-second sweep runs its checks one after the other with no limit for each: one slow check delays all the others, the liveness check too (CH19-3, CH19-8, CH01-3, CH08-3, CH08-8, CH10-3, CH13-8, CH18-3, CH06-8, CH02-3); the 07:10 audit has 10 minutes, measured with three boxes only (CH11-3, CH11-8). On the box: the interval jobs have no limit of their own (CC11-3, CC12-3, CC13-3, CC14-3, CC15-3, CC16-3, CC20-3, CA06-3, CA10-3, CA10-8, CA13-8, CA15-3, CA19-3). Every measured duration is from a small test store. | OPEN — filed 2026-11-11 (clock list) | — | Hub: log each sweep step's duration and alarm when one sweep takes longer than its interval; then read a week of numbers and set limits. Box: the same for the controller's interval jobs. | CC |
| **R-1393** | Hub & operator | P4 | **For about 8 minutes after a hub restart its database statements take 1 to 32 seconds; the System page took 12.7 s and a box's enrolment 11.4 s.** Measured 2026-11-14 at the 0.181.0 deploy (`audits/release-1-dates-2026-11-14/hub/05-watch.txt`): 24 `slow statement` lines from 09:58 to 10:06 UTC, none in the 20 minutes after. The database file is 678 MB; the slow ones read `host_reports`, `os_reports` and `events` (whole report JSON per row). One 32-second statement ran while CC copied the database out of the pod for a read (CC's own load). No report was lost that CC could see; not measured: whether a box's report timed out in that window. | OPEN — filed 2026-11-14 | — | measure a restart with no copy running; then decide: an index or a shorter keep on `host_reports`/`os_reports`, or accept | CC |
+131
View File
@@ -0,0 +1,131 @@
# A household or a tester leaves — the runbook
> **Status: written 2026-11-14 from source and from one real use; NOT run as a whole.** Operator ruling D74
> (`09-update-architecture.md` §3 decision 294, R-1375). Each step says **PROVEN** (with where) or **NOT PROVEN**.
> A step that is NOT PROVEN is not a wrong step: nobody has seen it work. Read its line before you press.
> Source: the endings list (`audits/endings-list-2026-11-10/TABLE.md`, endings EH00 to EH06) and the one real
> retirement (`audits/RETIRE-peti-2026-09-25.md`). The design of what the hub removes is `05-hub-architecture.md` §14.
## What this is, and what it is not
A household leaves Felhom, or a tester stops testing. You remove what Felhom holds of them: in the hub, at the
off-site provider, on ep0, and at Cloudflare. **There is no single button.** The hub has five doors; three are
buttons, one is a tick box, one is by hand.
It is NOT for a household that starts again on a new machine. For that, use RESET only (door 2) and stop.
## Before you start — four questions
1. **Does the household want its data?** The box holds the data. The hub holds none of it. Tell the household to
copy what it wants off the box (the file manager, or each app's own export) BEFORE you start. After door 3 the
off-site copies are gone and there is no undo. **NOT PROVEN:** no text for the household exists (capability E16;
R-810). Write to them by hand.
2. **Is the box still on?** An ONLINE box refuses doors 1 and 3. Ask the household to switch the box off, or wait
until the hub shows it as down (a silent box can be deleted after 6 minutes). PROVEN in tests
(`hub/internal/web/hosts.go`, the 409 gate; `customer_delete_test.go` GatesFailClosed).
3. **Was the off-site tier ever ticked for this household, and is it ticked now?** Read the household's config page.
If it was ticked once and is unticked now, see „The untick trap” below before door 3.
4. **Who is it?** Write the customer id and the host id on paper. You type both.
## The five doors, in the order you use them
| # | Door | Where | What it ends | Ending |
|---|---|---|---|---|
| 1 | Delete the box | hub → Hosts → the box → Delete | the box's record | EH00 |
| 2 | RESET the household | hub → the household → Reset | everything the box made, at the hub and outside; the household's record stays | EH01 |
| 3 | Delete the household | hub → the household → Delete customer… | doors 1 and 2 for every box, then the record itself | EH02 |
| 4 | Untick a tier | hub → the household → Edit | the tier is no longer offered to the box; NOTHING is removed | EH06 |
| 5 | Cloudflare, by hand | Cloudflare's dashboard | the tunnel, the DNS records, the API token | EH05 |
Doors 2 and 3 open two more by themselves: the off-site account (EH03) and the copies on ep0 (EH04).
**For a household that leaves: door 3, then door 5.** Door 3 does doors 1 and 2 for you.
**For a tester who stops: the same.** There is no other path for a tester (the endings list, „a tester ends”).
## The steps
### Step 1 — the household copies its data off the box. NOT PROVEN (no product path; by hand).
### Step 2 — the box is off, or silent for 6 minutes.
PROVEN in tests: an online box refuses (`TestCustomerDelete…GatesFailClosed`). NOT PROVEN on a real leaving
household.
### Step 3 — read the preview. hub → the household → Delete customer…
The dialog shows what will go: the boxes, the off-site account, the residue count, and the Cloudflare objects you
must remove by hand. **Read the off-site line.** It must name an account when the household ever had the off-site
tier. If it says „none” and you know there was one: STOP, go to „The untick trap”.
PROVEN: the preview was read at the one real use (`RETIRE-peti-2026-09-25.md` A3 step 3: 0 hosts, the sub-account,
residue 1,519). The Cloudflare line: PROVEN in a test only (`TestR688_PreviewNamesCloudflareByHand`).
### Step 4 — tick the three boxes, type the customer id, press.
The hub then runs, in this order: the journal entry → every box deleted (door 1) → the off-site account taken down
→ the copies on ep0 taken down → the claim reset → the hub's rows purged → the record and ALL key packages purged.
- Door 1 for each box: PROVEN in tests (`TestDeleteHost_CascadeAndIsolation`); a real box delete is routine
(Tester4, 2026-11-11 and 11-14, by the operator).
- The off-site account: **PROVEN once on a real account** (`RETIRE-peti-2026-09-25.md`: sub-account 269130 removed,
the pool's other accounts and their sizes unchanged). That account held no backup. **NOT PROVEN: an account that
holds a real backup.** The hub empties the folder through the account's own login first, then deletes the account;
if the emptying fails the account is KEPT and the step fails (R-32) — then press again.
- The copies on ep0: **NOT PROVEN on real copies** (at the one real use there was nothing on ep0: `existed=false`).
Tests only (`customer_delete_test.go`). The second copy of ep0 on DooPlex is NOT removed by this and was not
measured (`runbooks/ep0-datastore-copy.md`).
- The purge of the hub's rows: PROVEN once (482 reports, 1,036 telemetry rows, 1 log tail gone; journal #20).
**Known gap, R-1376:** rows in 13 tables stay (the box's update settings and reports, its kernel notices, the
off-site key and window rows, the abandon requests). Ruled to be removed (D73, `09` §3 decision 293); read the
row for whether that is released — until then they stay.
### Step 5 — if a leg fails: press again.
The dialog offers Resume. A leg that is done is not done again. Nothing is half forgotten: the journal names the leg.
PROVEN in tests (`…ResumesAfterExternalFailure`). NOT PROVEN live (`_recovery-inventory-2026-07-28.md`: „the
end-to-end live leg is not run”).
### Step 6 — check the provider by hand. 5 minutes.
At the off-site provider's page: the household's sub-account is gone; the others are there; the pool's size did not
jump. PROVEN as a check (it is what `RETIRE-peti-2026-09-25.md` A4 did). **Always do it when the journal says
`skipped_no_config`** — the hub could not know the account then.
### Step 7 — door 5, Cloudflare, by hand. NOT PROVEN (never done; no code does it; the operator accepted that, R-688).
From the dialog's list: 1. delete the tunnel; 2. delete the DNS records that point at it; 3. delete the household's
API token; 4. if the zone was added only for this household, remove the zone or leave it to the household (it is
THEIR domain — ask them). If you do nothing: the names point at a tunnel that no longer runs; a visitor sees
Cloudflare's error page. No data is exposed.
### Step 8 — what stays, on purpose. Do not „clean” these.
- The audit trail in the hub: the delete and reset journals, the host-deletion records, the events and the
notification log. The last two go by themselves after one year (R-901, PROVEN in a test).
- The hub's own database snapshots: they hold the household's rows for up to 8 weeks on ep0. That is the only way
back, and it is a retention fact you must be able to state to the household.
- The household's box and its drives: theirs. Felhom removes nothing on the box. If the household wants the box
wiped, that is a new install by them, or the box's own „factory” path — NOT PROVEN as part of leaving.
### Step 9 — a tester only: what to collect. NOT PROVEN (no list existed; this is the first).
Before step 4, ask the tester for: what did not work (in their words); whether they want to come back later (then
use RESET, door 2, and keep the record); the box's hub page saved as a file if a finding is open in the register.
After step 7: remove the tester from `runbooks/target-selection.md` (as Peti's entry was), and from `STATUS.md` if
named there.
## The untick trap (door 4) — read this if the off-site tier was unticked
Unticking the off-site tier removes nothing at the provider. **Until the fix is released, it also made the hub
forget the account:** a later RESET or delete then skipped it, and the account stayed, paid for, with the
household's copies in it, and no record in the hub (R-1375, endings EH06 and EH01).
- **Fix built, not released (hub `main`, 2026-11-14, D71):** the hub keeps the account's record after an untick, and
RESET and delete take the account down when a record exists.
- **Until that release, and for any household unticked before it:** before door 3, tick the off-site tier again and
save (this is ending EH06's own undo and gives the record back), THEN do door 3. NOT PROVEN on a real household.
- **If the record is already lost** (unticked, then RESET or deleted, before the fix): the account has the label
`felhom-customer=<customer id>` at the provider. Find it by the label, never by its number, and remove it by hand
as `RETIRE-peti-2026-09-25.md` A3 did. PROVEN once (that record).
## The second off-site tier (the whole-box copies on ep0)
Switching it off keeps the household's place and copies on ep0, on purpose (PROVEN in a test,
`TestPBSDR_DisableKeepsTenancy`). The hub finds them by the customer id, so door 2 and door 3 take them down whether
the tier is ticked or not. Nothing to do by hand here.
## What is not in this runbook, and why
- A mail to the household that says what was removed. It does not exist (R-810). Write by hand.
- A refund or an invoice. Money is the operator's.
- A wipe of the household's box. Not Felhom's to do.
+9
View File
@@ -3,6 +3,15 @@
- `bundlefiles`: the line „bundle X holds the same files as the vouched Y” is logged once per pair (INFO), not at
every view of the System page (it was 6 DEBUG lines per view, seen at the 0.181.0 deploy). No behaviour change.
- **Release 3 („the hub tells you”), built 2026-11-14 night, NOT deployed. Every new alarm is operator-only.**
- **The record of an off-site account outlives an untick (R-1375 part 1, D71, `09` §3 decision 291; `05` §14.1).**
The config form rebuilds the whole config at a save, so unticking the off-site tier dropped the `offsite` block and
the hub forgot the account; a later RESET or delete skipped it. Now `offsite.KeepAccountRecord` leaves
`offsite_account` (type, user, host, time) in the config and carries it through later saves; `web.offsiteChoice`
reads the tick OR the record, so RESET, delete and both previews find the account; the RESET's descriptor leg drops
the record with the account; the config form says that the account still exists. **An untick alone takes nothing
down** — pinned first. Tests `TestD71_*` (the real save and RESET over `hetznerapi.Fake`),
`TestKeepAccountRecord_FourCases`; 9 mutations seen red (one stayed green at first: the table had no case for it;
the case was added and the mutation done again).
- **Four silent failures tell the operator (R-1383, D78, `09` §3 decision 299; `05` §16.6).** No new table.
(1) `hub_daily_job_skipped`: `monitor.DailyLedger` keeps across a restart when `deadline-check` and
`offsite-key-audit` last ran; a slot that passed while the hub was down is told once at the next start (the check
@@ -0,0 +1,51 @@
package offsite
import (
"strings"
"testing"
"time"
)
// D71 (R-1375): the four cases of KeepAccountRecord, as a table. The web tests (TestD71_*) prove the consequence
// through the real save and the real RESET; this pins the rule itself.
func TestKeepAccountRecord_FourCases(t *testing.T) {
now := time.Date(2026, 11, 14, 20, 0, 0, 0, time.UTC)
const (
provisioned = `{"git":{"username":"x"},"offsite":{"enabled":true,"type":"shared","host":"u1-sub3.example","user":"u1-sub3","port":23}}`
choiceOnly = `{"offsite":{"enabled":true,"type":"shared","quota_gb":100}}` // ticked, never provisioned (or after a RESET)
withRecord = `{"offsite_account":{"type":"shared","user":"u1-sub3","host":"u1-sub3.example","unticked_at":"2026-11-01T00:00:00Z"}}`
unticked = `{"git":{"username":"x"}}`
)
for _, c := range []struct {
name, prev, next string
wantUser, wantAt string // "" = no record
keepsOther string
}{
{"ticked and provisioned, then unticked: a record from the old descriptor", provisioned, unticked, "u1-sub3", "2026-11-14T20:00:00Z", `"git"`},
{"a record, still unticked: carried unchanged (the form does not know it)", withRecord, unticked, "u1-sub3", "2026-11-01T00:00:00Z", `"git"`},
{"ticked again: no record", withRecord, provisioned, "", "", `"offsite"`},
{"a config that carries BOTH a ticked tier and a record: the record goes (the descriptor is the record)", withRecord,
`{"offsite":{"enabled":true,"type":"shared","user":"u1-sub3"},"offsite_account":{"type":"shared","user":"u1-sub3"}}`, "", "", `"offsite"`},
{"ticked, never provisioned, then unticked: no account was made, no record", choiceOnly, unticked, "", "", `"git"`},
{"never ticked: nothing", `{}`, unticked, "", "", `"git"`},
{"an empty previous config", ``, `{}`, "", "", ``},
} {
out, err := KeepAccountRecord(c.prev, c.next, now)
if err != nil {
t.Fatalf("%s: %v", c.name, err)
}
rec := ReadAccountRecord(out)
switch {
case c.wantUser == "" && rec != nil:
t.Errorf("%s: a record %+v, want none (%s)", c.name, rec, out)
case c.wantUser != "" && (rec == nil || rec.User != c.wantUser || rec.UntickedAt != c.wantAt || rec.Type != "shared"):
t.Errorf("%s: record %+v, want user %s unticked_at %s (%s)", c.name, rec, c.wantUser, c.wantAt, out)
}
if c.keepsOther != "" && !strings.Contains(out, c.keepsOther) {
t.Errorf("%s: the other keys were lost: %s", c.name, out)
}
}
if _, err := KeepAccountRecord(provisioned, `{not json`, now); err == nil {
t.Error("a config that cannot be read must be an error (the save is then refused), not a silent loss of the record")
}
}
+93
View File
@@ -514,6 +514,8 @@ func ClearProvisionedDescriptor(configJSON string) (string, error) {
if err != nil {
return "", err
}
// D71: the record of an unticked tier's account dies with the account too (RESET took it down before this leg).
configJSON = dropAccountRecord(configJSON)
if cur == nil {
return configJSON, nil // no offsite block — nothing to clear
}
@@ -521,6 +523,97 @@ func ClearProvisionedDescriptor(configJSON string) (string, error) {
return MergeDescriptor(configJSON, cleared)
}
func dropAccountRecord(configJSON string) string {
obj := map[string]json.RawMessage{}
if json.Unmarshal([]byte(configJSON), &obj) != nil {
return configJSON
}
if _, ok := obj[AccountRecordKey]; !ok {
return configJSON
}
delete(obj, AccountRecordKey)
out, err := json.Marshal(obj)
if err != nil {
return configJSON
}
return string(out)
}
// AccountRecordKey is the ConfigJSON key that holds the record of an off-site account whose tier was UNTICKED
// (D71, `09` §3 decision 291, R-1375; `06-offsite-connectivity.md`). The box does not read it.
const AccountRecordKey = "offsite_account"
// AccountRecord says: an off-site account for this household still exists at the provider although the tier is
// switched off. Before it, unticking the tier dropped the whole `offsite` block (the form rebuilds ConfigJSON), so
// the hub forgot the account: a later RESET or delete skipped it and the account stayed, paid for, with the
// household's copies in it and no record anywhere (endings EH06-4, EH01-4). Non-secret.
type AccountRecord struct {
Type string `json:"type,omitempty"` // shared | dedicated — what Deprovision needs
User string `json:"user,omitempty"`
Host string `json:"host,omitempty"`
UntickedAt string `json:"unticked_at,omitempty"` // RFC3339 UTC
}
// ReadAccountRecord returns the record, or nil when there is none (or it cannot be read).
func ReadAccountRecord(configJSON string) *AccountRecord {
obj := map[string]json.RawMessage{}
if json.Unmarshal([]byte(configJSON), &obj) != nil {
return nil
}
raw, ok := obj[AccountRecordKey]
if !ok {
return nil
}
var r AccountRecord
if json.Unmarshal(raw, &r) != nil {
return nil
}
return &r
}
// KeepAccountRecord is called on every save of a household's config, with the config as it WAS and as the form
// built it. It returns the new config with the account record carried:
// - the tier is ticked in the new config: no record (the descriptor itself is the record);
// - the tier was ticked AND PROVISIONED before and is unticked now: a record is written from the old descriptor;
// - a record was there before and the tier is still unticked: the record is carried (the form does not know it);
// - else: nothing — a tier that was ticked and never provisioned made no account.
//
// It calls no provider and removes nothing: AN UNTICK ALONE NEVER TAKES AN ACCOUNT OR ITS DATA DOWN. Only RESET and
// delete do (web.commitCustomerReset), and they read this record. Pinned by TestKeepAccountRecord_* and
// web TestD71_*.
func KeepAccountRecord(prevJSON, newJSON string, now time.Time) (string, error) {
obj := map[string]json.RawMessage{}
if strings.TrimSpace(newJSON) != "" {
if err := json.Unmarshal([]byte(newJSON), &obj); err != nil {
return "", fmt.Errorf("offsite: parse config_json: %w", err)
}
}
newD, err := ReadDescriptor(newJSON)
if err != nil {
return "", err
}
if newD != nil && newD.Enabled {
delete(obj, AccountRecordKey)
} else {
var rec *AccountRecord
if prevD, _ := ReadDescriptor(prevJSON); prevD != nil && prevD.Enabled && (prevD.User != "" || prevD.Host != "") {
rec = &AccountRecord{Type: prevD.Type, User: prevD.User, Host: prevD.Host, UntickedAt: now.UTC().Format(time.RFC3339)}
} else if prevRec := ReadAccountRecord(prevJSON); prevRec != nil {
rec = prevRec
}
if rec == nil {
return newJSON, nil
}
rb, _ := json.Marshal(rec)
obj[AccountRecordKey] = rb
}
out, err := json.Marshal(obj)
if err != nil {
return "", err
}
return string(out), nil
}
// ReadDescriptor extracts the non-secret offsite Descriptor from a customer's ConfigJSON, or (nil, nil)
// when there is no offsite block (never provisioned). This is the AUTHORITATIVE tier/quota source — the
// pool-box Σ(quota) aggregate (v0.64.0, R-5) and the RESET clear both read through it, NEVER the report
+13
View File
@@ -916,6 +916,7 @@ func (s *Server) handleConfigUpdate(w http.ResponseWriter, r *http.Request, cust
}
}
prevConfigJSON := cfg.ConfigJSON // D71: the form rebuilds ConfigJSON; the account's record must outlive an untick
cfg.ConfigJSON = buildConfigJSON(r)
if err := s.applyOffsite(r.Context(), r, cfg); errors.Is(err, offsite.ErrProvisionInProgress) {
@@ -928,6 +929,18 @@ func (s *Server) handleConfigUpdate(w http.ResponseWriter, r *http.Request, cust
http.Error(w, "Offsite provisioning failed: "+err.Error(), http.StatusBadGateway)
return
}
// D71 (R-1375): unticking the off-site tier takes NOTHING down — and the hub now keeps the record of the account,
// so a later RESET or delete finds it. Fail-closed: a config we cannot carry the record into is not saved.
if kept, kerr := offsite.KeepAccountRecord(prevConfigJSON, cfg.ConfigJSON, time.Now()); kerr != nil {
s.logger.Printf("[ERROR] offsite account record for %s: %v", customerID, kerr)
http.Error(w, "Internal error", http.StatusInternalServerError)
return
} else {
if rec := offsite.ReadAccountRecord(kept); rec != nil && offsite.ReadAccountRecord(prevConfigJSON) == nil {
s.logger.Printf("[WARN] offsite tier UNTICKED for %s: nothing was removed at the provider; the account (%s, %s) stays and its record is kept — RESET or delete takes it down", customerID, rec.Type, rec.User)
}
cfg.ConfigJSON = kept
}
// PBS DR tier (fail-closed; idempotent on an already-provisioned descriptor — no re-key,
// no second secret, no spurious generation bump).
+10
View File
@@ -23,6 +23,9 @@ import (
// offsiteChoice reads the customer's offsite tier selection out of config_json (the same shape the
// re-issue/freeze handlers read). enabled=false means no offsite leg to run.
// offsiteChoice answers „is there an off-site account of this household to take down, and of which type?” — the tier
// is ticked, OR it was unticked and the account's record was kept (offsite.KeepAccountRecord). Read by the RESET
// commit and by both previews. Pinned by TestD71_*.
func offsiteChoice(configJSON string) (enabled bool, typ string) {
var o struct {
Offsite struct {
@@ -31,6 +34,13 @@ func offsiteChoice(configJSON string) (enabled bool, typ string) {
} `json:"offsite"`
}
_ = json.Unmarshal([]byte(configJSON), &o)
if !o.Offsite.Enabled {
// D71 (`09` §3 decision 291, R-1375): the tier is unticked, but the hub kept the record of the account it
// made — RESET and delete take that account down too, and their previews name it. An untick alone never does.
if rec := offsite.ReadAccountRecord(configJSON); rec != nil {
return true, rec.Type
}
}
return o.Offsite.Enabled, o.Offsite.Type
}
@@ -0,0 +1,174 @@
package web
import (
"context"
"encoding/json"
"io"
"log"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-hub/internal/hetznerapi"
"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// d71Server: the hub's web server over the repo's FAKE provider (hetznerapi.Fake — no provider is called). purges
// counts the calls to the shared folder's purge, the step that destroys a household's off-site copies.
func d71Server(t *testing.T) (*Server, *store.Store, *hetznerapi.Fake, *int) {
t.Helper()
s, st := newTestServer(t)
api := hetznerapi.NewFake()
purges := 0
s.SetOffsiteProvisioner(&offsite.Provisioner{
API: api, Store: st, Scanner: webTestScanner{}, PoolBoxID: 611714, Location: "fsn1",
Logger: log.New(io.Discard, "", 0),
PurgeShared: func(context.Context, string) error { purges++; return nil },
})
s.SetTenantSync(&fakeTenancy{})
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "acme", CustomerName: "Acme", Domain: "acme.hu",
Email: "a@example.com", RetrievalPassword: "pw", APIKey: "k", ConfigJSON: "{}"}); err != nil {
t.Fatal(err)
}
return s, st, api, &purges
}
func d71Save(t *testing.T, s *Server, form string) {
t.Helper()
w := httptest.NewRecorder()
s.handleConfigUpdate(w, postForm("/configs/acme/edit", "customer_name=Acme&domain=acme.hu&dr_tier=on"+form), "acme")
if w.Code != http.StatusSeeOther {
t.Fatalf("save (%s): %d %s", form, w.Code, w.Body.String())
}
}
func d71Accounts(t *testing.T, api *hetznerapi.Fake) int {
t.Helper()
subs, err := api.ListSubaccounts(context.Background(), 611714, "")
if err != nil {
t.Fatal(err)
}
return len(subs)
}
const d71Tick = "&offsite_enabled=on&offsite_type=shared&offsite_quota_gb=50"
// D71 (`09` §3 decision 291, R-1375). THE WRONG CASE, pinned first: an untick alone must NEVER take an account or
// its data down. Then the fix: the hub keeps the account's record through the untick (and through later saves — the
// form rebuilds the whole config), so a later RESET finds the account and takes it down; before, RESET skipped it and
// the account stayed at the provider, paid for, with no record in the hub.
// COMPANION RED-PROOFS (observed, audits/night-build-2026-11-14/B/redproof-hub-offsite-record.txt).
func TestD71_UntickKeepsTheAccountAndItsRecord_ResetTakesItDown(t *testing.T) {
s, st, api, purges := d71Server(t)
d71Save(t, s, d71Tick)
if n := d71Accounts(t, api); n != 1 {
t.Fatalf("setup: %d account(s) at the fake provider after the tick, want 1", n)
}
cfg, _ := st.GetCustomerConfig("acme")
d, _ := offsite.ReadDescriptor(cfg.ConfigJSON)
if d == nil || d.User == "" {
t.Fatalf("setup: no provisioned descriptor: %s", cfg.ConfigJSON)
}
// UNTICK — and two more saves after it
d71Save(t, s, "")
d71Save(t, s, "")
d71Save(t, s, "&debug_mode=on")
if n := d71Accounts(t, api); n != 1 || *purges != 0 {
t.Fatalf("AN UNTICK TOOK SOMETHING DOWN: %d account(s) left (want 1), %d purge call(s) (want 0)", n, *purges)
}
cfg, _ = st.GetCustomerConfig("acme")
if nd, _ := offsite.ReadDescriptor(cfg.ConfigJSON); nd != nil && nd.Enabled {
t.Fatalf("the tier is still enabled after the untick: %s", cfg.ConfigJSON)
}
rec := offsite.ReadAccountRecord(cfg.ConfigJSON)
if rec == nil || rec.User != d.User || rec.Type != "shared" || rec.UntickedAt == "" {
t.Fatalf("the account's record did not survive the untick and two saves: %+v in %s", rec, cfg.ConfigJSON)
}
// the box is handed the tier as OFF: the record is not an `offsite` block
var box map[string]json.RawMessage
_ = json.Unmarshal([]byte(cfg.ConfigJSON), &box)
if _, has := box["offsite"]; has {
t.Errorf("an unticked tier still carries an `offsite` block the box would read: %s", cfg.ConfigJSON)
}
// both previews name the account
for _, path := range []string{"reset", "delete"} {
rr := httptest.NewRecorder()
req := httptest.NewRequest("GET", "/configs/acme/"+path, nil)
if path == "reset" {
s.handleCustomerResetPreview(rr, req, "acme")
} else {
s.handleCustomerDeletePreview(rr, req, "acme")
}
if !strings.Contains(rr.Body.String(), `"offsite_enabled":true`) || !strings.Contains(rr.Body.String(), d.User) {
t.Errorf("the %s preview does not name the kept account %s: %s", path, d.User, rr.Body.String())
}
}
// the form says it
page := httptest.NewRecorder()
s.renderConfigForm(page, httptest.NewRequest("GET", "/configs/acme/edit", nil), false, cfg, nil, "")
if !strings.Contains(page.Body.String(), "data-offsite-account-kept") || !strings.Contains(page.Body.String(), d.User) {
t.Errorf("the config form does not say that the account still exists")
}
// RESET: the account goes, its folder is purged first, and the record goes with it
rr := postReset(t, s, "acme", url.Values{"confirm_id": {"acme"}})
if rr.Code != http.StatusSeeOther {
t.Fatalf("reset: %d %s", rr.Code, rr.Body.String())
}
if n := d71Accounts(t, api); n != 0 || *purges != 1 {
t.Fatalf("after the RESET: %d account(s) left (want 0), %d purge(s) (want 1) — the unticked account was skipped", n, *purges)
}
cr, _ := st.LatestCustomerReset("acme")
if cr == nil || cr.Legs["hetzner"] != "ok" {
t.Fatalf("journal: %+v, want the hetzner leg ok (not skipped)", cr)
}
cfg, _ = st.GetCustomerConfig("acme")
if offsite.ReadAccountRecord(cfg.ConfigJSON) != nil {
t.Errorf("the record outlived the account: %s", cfg.ConfigJSON)
}
}
// Ticking again takes the same account back into use (the provider is asked by the household's label, so no second
// account is made) and the record is gone: the descriptor is the record again.
func TestD71_TickAgainUsesTheSameAccount(t *testing.T) {
s, st, api, purges := d71Server(t)
d71Save(t, s, d71Tick)
cfg, _ := st.GetCustomerConfig("acme")
first, _ := offsite.ReadDescriptor(cfg.ConfigJSON)
d71Save(t, s, "")
d71Save(t, s, d71Tick)
cfg, _ = st.GetCustomerConfig("acme")
again, _ := offsite.ReadDescriptor(cfg.ConfigJSON)
if again == nil || !again.Enabled || again.User != first.User {
t.Fatalf("after ticking again: %+v, want the first account %s", again, first.User)
}
if offsite.ReadAccountRecord(cfg.ConfigJSON) != nil {
t.Errorf("a ticked tier still carries the untick record: %s", cfg.ConfigJSON)
}
if n := d71Accounts(t, api); n != 1 || *purges != 0 {
t.Fatalf("accounts=%d purges=%d, want 1 and 0", n, *purges)
}
}
// A household that never had the tier gets no record, and a RESET then calls no provider (as before).
func TestD71_NeverTickedMakesNoRecord(t *testing.T) {
s, st, api, purges := d71Server(t)
d71Save(t, s, "")
d71Save(t, s, "")
cfg, _ := st.GetCustomerConfig("acme")
if offsite.ReadAccountRecord(cfg.ConfigJSON) != nil {
t.Fatalf("a record for an account that never existed: %s", cfg.ConfigJSON)
}
if rr := postReset(t, s, "acme", url.Values{"confirm_id": {"acme"}}); rr.Code != http.StatusSeeOther {
t.Fatalf("reset: %d", rr.Code)
}
cr, _ := st.LatestCustomerReset("acme")
if cr == nil || cr.Legs["hetzner"] != "skipped" || d71Accounts(t, api) != 0 || *purges != 0 {
t.Fatalf("journal %+v, purges %d — want the hetzner leg skipped and nothing called", cr, *purges)
}
_ = http.StatusOK
}
@@ -97,13 +97,14 @@
</div>
</details>
<details class="card" {{if index .Overrides "offsite"}}open{{end}}>
<details class="card" {{if or (index .Overrides "offsite") (index .Overrides "offsite_account")}}open{{end}}>
<summary><h2 style="display:inline">Offsite backup</h2></summary>
<div class="form-grid" style="margin-top: 1rem;">
<div class="form-group">
<label><input type="checkbox" name="offsite_enabled"
{{with .Overrides}}{{with index . "offsite"}}{{if index . "enabled"}}checked{{end}}{{end}}{{end}}>
Enable offsite (provisions a Hetzner Storage Box on save)</label>
{{with index .Overrides "offsite_account"}}<p class="form-hint" data-offsite-account-kept>The off-site tier is OFF, and this household's account still exists at the provider: <strong>{{index . "user"}}</strong> ({{index . "type"}}), unticked {{index . "unticked_at"}}. Unticking removed nothing. A RESET or „Delete customer…” takes the account and its copies down. Tick again to use it.</p>{{end}}
</div>
<div class="form-group">
<label for="offsite_type">Type</label>