hub: the off-site copies of a removed app end after 30 days — the hub agrees first, the allowance is beside the cap, the household's mail (R-1374 part 1, D72, D84)

Built, not deployed. `07-backup-architecture.md` §8.2f parts 3-4 and decision 309.

- store: the hub's own record of which apps a box runs, by the hub's clock
  (offsite_app_presence, fed by every controller report); it goes with the household.
- offsitekeys: window-open takes the box's due end records and answers
  ended_allowed / ended_refused. An app is agreed only after 30 days of absence
  in the hub's record; an id only when the hub's own listing holds it, uploaded
  before that absence. The cap (max_remove) is unchanged; the close keeps only
  agreed ids from removed_for_ended_apps and allows them beside the cap.
- The 07:10 audit explains the ended apps' vanished files by name and still
  raises for every other vanished file.
- notify: offsite_end_scheduled is a household receipt — one mail per press to
  the household's own address, whatever its switches; the operator is not mailed.
- Two operator-only hub records: offsite_ended_app_removed (info),
  offsite_ended_app_refused (warning).

Tests TestD72_*, TestD84_*; 44 mutations seen red, 5 after a redo
(documentation/audits/build-release-4-2026-11-15/A/redproof-hub-*.txt).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-15 13:26:13 +01:00
1 parent 552e19cd3d
commit 01a106b78c
33 files changed
+1834 -24

No files matched your search

+2
View File
@@ -77,6 +77,8 @@
| `offsitekeys.Registrar` (`Install` / `Confirm` / `Audit` / `OpenWindow` / `CloseWindow` / `MoveAside`) | hub/internal/offsitekeys/offsitekeys.go | `(ctx, Target, password, …)` | EVERY write to a sub-account's `.ssh/authorized_keys` and every repo move-aside | **The only writer of that file, and the only deleter on a sub-account (`DeleteSetAside`: `<repo>.orphaned-*` only, decision 74).** `read()` is read-only (R-827). Uses the provider's port-23 restricted shell (`dd of=` takes stdin, `mv` overwrites, `test` does NOT exist — measured); an unpinned line is a deletion route and is dropped on every install; the window line goes FIRST (first match wins). Never `rm`. |
| `offsitekeys.Service` (`RegisterKey`, `ConfirmKey`, `AuditAll`, `OpenWindowFor`, `CloseWindowFor`, `SweepExpiredWindows`) | hub/internal/offsitekeys/service.go | — | Binding the registrar to the store, descriptor and operator events | The box-facing API (`/api/v1/offsite/register-key…`) answers with NO credential — pinned by `TestOffsiteKeyEndpoints_AuthAndNoPasswordInAnyResponse`. |
| `offsitekeys` snapshot watch + `store.Get/SetOffsiteSnapshotListing` / `OffsiteRunTimesSince` (v0.158.0, R-895 slice 2) | hub/internal/offsitekeys/snapshot_watch.go, hub/internal/store/offsite_snapshots.go | — | Judging the 07:10 `snapshots/` listing against yesterday's (baseline in `hub_settings`) and the box's reported runs | Raises `offsite_snapshots_vanished` / `_unexplained` — operator-only and hub-origin (NOT in `allowedEventTypes`). The first listing is a baseline; a move-aside or RESET drops it. |
| `store.NoteAppList` / `AppPresences` + `offsitekeys.agreeEnded` (D72/D84, `07` §8.2f) | hub/internal/store/offsite_ended.go, hub/internal/offsitekeys/ended.go | `NoteAppList(customerID, deployed, claimed, now)`; `agreeEnded(now, presence, listing, haveListing, claims)` | The hub's OWN record of when an app left a box's app list (fed by `handleReport`), and the decision which snapshot ids of ended apps a window may remove outside its cap | **The box's dates decide nothing.** An id is agreed only when its app has been absent 30 days by the hub's clock AND the hub's own listing shows the file older than that absence. `WindowCredit.CapExplains` is the cap's part, `EndedIDs` the agreed ids — a reader that explains ids by name must not also spend `Explains`. |
| `notify.householdReceiptEvents` (D84) | hub/internal/notify/dispatcher.go | register | A mail that confirms the household's own press (one type: `offsite_end_scheduled`) | Bypasses the household's group switches and is keyed per press (details `subject`); the operator is not mailed. **Every new entry needs its own ruling** — pinned by `TestD84_Receipt_OpensNothingElse`. |
| `accesslog.New` / `Middleware` / `Mark` (v0.158.0, R-1140) | hub/internal/accesslog/accesslog.go | `Mark(ctx, kind)` | The hub's own access log (`<data_dir>/access-log/`, 90 days) and marking the identity KIND an auth check established | Never log a value: the path only (no query; `/bind/<token>` and 32+-char segments redacted). A new auth path calls `Mark` with its kind. A logging failure never fails a request. |
| the timed SQLite driver `sqlite-timed` + `SetSlowStatementThreshold` (v0.178.0, R-1346) | hub/internal/store/timeddriver.go | `store.New` opens it — nothing to call | Names a statement over 1 s: `[WARN] slow statement: <ms> (over 1s): <SQL>` | The SQL text only, never a bound value. Put a heavy statement's SQL in a const and pin its plan (`r1346_speed.go`, `TestR1346_HeavyStatementsUseTheirIndex`): on the hub's network disk a whole-table read is seconds to minutes. |
| `(*Store).SaveOneTimeSecret` / `OffsitePassword` / `SealLegacyOffsiteSecrets` | hub/internal/store/offsite_seal.go | — | Storing / reading the sub-account password | **Sealed AES-256-GCM; no key → refused (fail-closed).** Under `go test` every store gets a fixed key (`testing.Testing()`); production needs `OFFSITE_SECRET_KEY`. Never serve the value to a box. |
@@ -0,0 +1,29 @@
### internal/api/handler.go
== MUTATION: a report does not feed the hub's app record
RED on the mutant, GREEN on the real file
--- FAIL: TestD72_ReportFeedsTheHubsOwnAppRecord (0.05s)
d84_ended_test.go:41: immich is listed: map[]
== MUTATION: a pending end record does not start the hub's clock
RED on the mutant, GREEN on the real file
--- FAIL: TestD72_ReportFeedsTheHubsOwnAppRecord (0.05s)
d84_ended_test.go:46: oldapp must be absent since NOW by the hub's clock (the box's 2024 date is not read): {Present:false AbsentSince:0001-01-01 00:00:00 +0000 UTC} ok=false
== MUTATION: a report with no app list empties the record
RED on the mutant, GREEN on the real file
--- FAIL: TestD72_ReportFeedsTheHubsOwnAppRecord (0.05s)
/mnt/5_hdd/felhom.eu/git/felhom.eu/hub/internal/api/d84_ended_test.go:27 +0x23d
/mnt/5_hdd/felhom.eu/git/felhom.eu/hub/internal/api/d84_ended_test.go:50 +0x4b7
== MUTATION: a box cannot send the receipt (400)
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_EventTypes (0.00s)
d84_ended_test.go:79: offsite_end_scheduled must be accepted from a box (POST /event would answer 400)
4 mutations, 0 not a red-proof
### internal/api/offsite.go
== MUTATION: the window-open drops the box's claims
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_WindowOpenCarriesTheEndedClaims (0.04s)
d84_ended_test.go:72: the service did not receive the claim: []
1 mutations, 0 not a red-proof
@@ -0,0 +1,111 @@
### internal/offsitekeys/ended.go
== MUTATION: an installed app is agreed
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_AgreeEnded_TheHubsOwnRecordsDecide (0.00s)
ended_test.go:110: refusal for "vikunja" = "the hub's record of when the app left cannot be read", want it to say "lists the app as installed" (all: [{App:vikunja Reason:the hub's record of when the app left cannot be read} {App:
ended_test.go:110: refusal for "immich" = "the hub's record of when the app left cannot be read", want it to say "installed" (all: [{App:immich Reason:the hub's record of when the app left cannot be read} {App: Reason:1 snapshot i
--- FAIL: TestD84_Window_TheEndedAllowanceIsBesideTheCap (0.05s)
== MUTATION: the 30 days are not checked
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_AgreeEnded_TheHubsOwnRecordsDecide (0.00s)
ended_test.go:106: agreed ids = [a1b2c685], want [] (refused: [])
ended_test.go:106: agreed ids = [a1b2c685], want [] (refused: [])
ended_test.go:106: agreed ids = [a1b2c685], want [] (refused: [])
== MUTATION: an app with no hub record is not refused as such
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/offsitekeys [build failed]
internal/offsitekeys/ended.go:75:6: declared and not used: known
== MUTATION: an unreadable absence date reads as long ago
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_AgreeEnded_TheHubsOwnRecordsDecide (0.00s)
--- FAIL: TestD84_AgreeEnded_TheHubsOwnRecordsDecide/an_absence_whose_date_cannot_be_read_is_never_long_ago (0.00s)
ended_test.go:110: refusal for "vikunja" = "", want it to say "cannot be read" (all: [{App: Reason:1 snapshot id(s) refused: uploaded after the hub saw the app leave}])
== MUTATION: _shares and the marker are eligible
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_AgreeEnded_TheHubsOwnRecordsDecide (0.00s)
ended_test.go:110: refusal for "_shares" = "", want it to say "not an app" (all: [{App: Reason:2 snapshot id(s) refused: uploaded after the hub saw the app leave}])
== MUTATION: a snapshot also listed under a refused app is agreed
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_AgreeEnded_TheHubsOwnRecordsDecide (0.00s)
ended_test.go:106: agreed ids = [a1b2c685 a1b2c686], want [a1b2c685] (refused: [{App:immich Reason:the box's newest report lists the app as installed}])
== MUTATION: a file uploaded after the app left is agreed
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_AgreeEnded_TheHubsOwnRecordsDecide (0.00s)
ended_test.go:106: agreed ids = [a1b2c685 a1b2c68b], want [a1b2c685] (refused: [])
ended_test.go:106: agreed ids = [a1b2c685 a1b2c688], want [a1b2c685] (refused: [])
== MUTATION: a shared snapshot is measured against the FIRST app only
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_AgreeEnded_TheHubsOwnRecordsDecide (0.00s)
ended_test.go:106: agreed ids = [a1b2c685 a1b2c68b], want [a1b2c685] (refused: [])
== MUTATION: ids are agreed with no hub listing
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_AgreeEnded_TheHubsOwnRecordsDecide (0.00s)
ended_test.go:106: agreed ids = [a1b2c685], want [] (refused: [])
== MUTATION: a file with no upload time is agreed
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_AgreeEnded_TheHubsOwnRecordsDecide (0.00s)
ended_test.go:106: agreed ids = [a1b2c689 a1b2c68a], want [a1b2c68a] (refused: [])
== MUTATION: an id the hub's listing does not hold is not refused as such
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/offsitekeys [build failed]
internal/offsitekeys/ended.go:139:7: declared and not used: listed
== MUTATION: every refusal is told to the operator
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Window_ARefusalTellsTheOperatorOnlyWhenItIsClear (0.04s)
ended_test.go:250: the operator must hear kimai (2 days) and immich (installed), and not `nearly` (an hour early); heard [kimai nearly immich]
== MUTATION: an installed app's refusal is NOT told
*** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0)
13 mutations, 3 not a red-proof
### internal/offsitekeys/service.go
== MUTATION: the ended ids get no allowance of their own (they count under the cap)
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Window_TheEndedAllowanceIsBesideTheCap (0.05s)
ended_test.go:185: a fall of cap + agreed ended ids raised the drop alarm: [{typ:offsite_window_drop sev:error msg:Off-site clean-up window 1: the snapshot count fell 20 -> 4, more than a window may remove (10).}]
== MUTATION: the allowance takes the box's word, not the agreed ids
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Window_TheEndedAllowanceIsBesideTheCap (0.05s)
ended_test.go:206: a fall beyond the cap with NO agreed id must raise the drop alarm, got [{typ:offsite_ended_app_refused sev:warning msg:Off-site clean-up: the box of c1 asked to remove the off-site copies of the removed app "imm
== MUTATION: the grant does not carry what the hub agreed to
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Window_TheEndedAllowanceIsBesideTheCap (0.05s)
ended_test.go:173: agreed ids = [], want the six of vikunja
== MUTATION: the grant does not say what was refused
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Window_TheEndedAllowanceIsBesideTheCap (0.04s)
ended_test.go:176: the window must say that immich was refused, and why: []
== MUTATION: no per-app record at the close
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Window_TheEndedAllowanceIsBesideTheCap (0.05s)
ended_test.go:189: want one info record for vikunja with 6 snapshots: []
5 mutations, 0 not a red-proof
### internal/offsitekeys/snapshot_watch.go
== MUTATION: the audit spends the ended allowance on OTHER vanished files
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Audit_ExplainsEndedAppsAndStillRaisesForTheRest (0.09s)
ended_test.go:300: one file vanished outside the agreement: want it judged alone, got 0
== MUTATION: the audit does not explain the ended apps' files
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Audit_ExplainsEndedAppsAndStillRaisesForTheRest (0.04s)
ended_test.go:286: the ended apps' files were judged as vanished (3)
2 mutations, 0 not a red-proof
### internal/offsitekeys/ended.go — the three mutations done again
== MUTATION: REDO (the first mutant did not compile): an app with no hub record is not refused as such
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_AgreeEnded_TheHubsOwnRecordsDecide (0.00s)
ended_test.go:110: refusal for "ghost" = "the hub's record of when the app left cannot be read", want it to say "no record" (all: [{App:ghost Reason:the hub's record of when the app left cannot be read} {App: Reason:1 snapshot id(
== MUTATION: REDO (the first mutant did not compile): an id the hub's listing does not hold is not refused as such
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_AgreeEnded_TheHubsOwnRecordsDecide (0.00s)
ended_test.go:110: refusal for "" = "1 snapshot id(s) refused: no upload time in the hub's own listing", want it to say "not in the hub's own listing" (all: [{App: Reason:1 snapshot id(s) refused: no upload time in the hub's own l
== MUTATION: REDO (the first mutant was EQUAL to the real code: `p.Present` was redundant beside the zero date and was removed from the source): an installed app's refusal is NOT told
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Window_ARefusalTellsTheOperatorOnlyWhenItIsClear (0.04s)
ended_test.go:250: the operator must hear kimai (2 days) and immich (installed), and not `nearly` (an hour early); heard [kimai]
3 mutations, 0 not a red-proof
@@ -0,0 +1,44 @@
### internal/notify/dispatcher.go
== MUTATION: the receipt reads the household's switches
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Receipt_ReachesTheHouseholdWhateverItsSwitches (0.04s)
d84_receipt_test.go:43: mails = 0, want exactly one (to the household): []
--- FAIL: TestD84_Receipt_FollowsTheHouseholdsLanguage (0.04s)
d84_receipt_test.go:70: mails = 0: []
== MUTATION: the receipt's cooldown is per type, not per press
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Receipt_OncePerPress (0.04s)
d84_receipt_test.go:97: three presses gave 1 mails, want 3: [{to:customer@example.com subject:[Felhom] Információ: vikunja: a távoli másolatok törlésre kijelölve body:Kedves Ügyfél!
== MUTATION: no receipt branch: an info event mails nobody
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Receipt_ReachesTheHouseholdWhateverItsSwitches (0.04s)
d84_receipt_test.go:43: mails = 0, want exactly one (to the household): []
--- FAIL: TestD84_Receipt_FollowsTheHouseholdsLanguage (0.05s)
d84_receipt_test.go:70: mails = 0: []
== MUTATION: the receipt also mails the operator
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Receipt_ReachesTheHouseholdWhateverItsSwitches (0.04s)
d84_receipt_test.go:43: mails = 2, want exactly one (to the household): [{to:operator@felhom.eu subject:[Felhom] ⚠️ c1: offsite_end_scheduled body:Customer: c1
--- FAIL: TestD84_Receipt_FollowsTheHouseholdsLanguage (0.04s)
d84_receipt_test.go:70: mails = 2: [{to:operator@felhom.eu subject:[Felhom] ⚠️ c1: offsite_end_scheduled body:Customer: c1
== MUTATION: the hub-made records are not operator-only
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_EventTypes (0.00s)
d84_ended_test.go:89: offsite_ended_app_removed must be operator-only
5 mutations, 0 not a red-proof
### internal/notify/templates.go
== MUTATION: the receipt carries the raw details
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Receipt_ReachesTheHouseholdWhateverItsSwitches (0.04s)
d84_receipt_test.go:58: the body carries the raw details:
== MUTATION: the receipt's subject does not name the app
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_Receipt_ReachesTheHouseholdWhateverItsSwitches (0.04s)
d84_receipt_test.go:50: the subject does not name the app: "[Felhom] Információ: %s: a távoli másolatok törlésre kijelölve"
--- FAIL: TestD84_Receipt_FollowsTheHouseholdsLanguage (0.04s)
d84_receipt_test.go:74: not the English receipt:
2 mutations, 0 not a red-proof
@@ -0,0 +1,16 @@
== MUTATION: the window block never shows the removed-apps line
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_TheWindowBlockSaysWhatWasAgreedAndRefused (0.15s)
offsite_window_test.go:261: the window block lacks "id=\"window-ended\""
offsite_window_test.go:261: the window block lacks "3 snapshot(s) agreed outside the cap, 2 removed"
offsite_window_test.go:261: the window block lacks "<strong>refused</strong> immich: the box&#39;s newest report lists the app as installed"
== MUTATION: the window block does not say what was refused
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_TheWindowBlockSaysWhatWasAgreedAndRefused (0.16s)
offsite_window_test.go:261: the window block lacks "<strong>refused</strong> immich: the box&#39;s newest report lists the app as installed"
== MUTATION: the removed-apps line shows on every window
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_TheWindowBlockSaysWhatWasAgreedAndRefused (0.15s)
offsite_window_test.go:249: a window with no removed app must not render the removed-apps line
3 mutations, 0 not a red-proof
@@ -0,0 +1,61 @@
### internal/store/offsite_ended.go
== MUTATION: an absent app's date moves at every report
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/store [build failed]
internal/store/offsite_ended.go:92:11: declared and not used: p
== MUTATION: an install does not end the absence
RED on the mutant, GREEN on the real file
--- FAIL: TestD72_NoteAppList_TheHubsOwnClock (0.05s)
d84_ended_test.go:43: an install must end the absence: {Present:false AbsentSince:2026-11-01 04:00:00 +0000 UTC}
== MUTATION: a repeated claim moves the hub's date
RED on the mutant, GREEN on the real file
--- FAIL: TestD72_NoteAppList_TheHubsOwnClock (0.05s)
d84_ended_test.go:54: a claimed, never-seen app must be absent since the first claim (2026-11-04 11:00:00 +0000 UTC): {Present:false AbsentSince:2026-11-04 21:00:00 +0000 UTC} ok=true
== MUTATION: a claim makes a LISTED app absent
*** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0)
== MUTATION: the box's result stores ids the hub never agreed to
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_EndedRemoved_OnlyAgreedIdsAreStored (0.04s)
d84_ended_test.go:129: kept = [a1 a2 living zz] err=<nil>, want [a1 a2]
5 mutations, 2 not a red-proof
### internal/store/offsite_keys.go
== MUTATION: a closed window explains every AGREED id, not the ones the box named
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_WindowCredits_TheEndedAllowanceIsBesideTheCap (0.04s)
d84_ended_test.go:93: a: want cap 9 + ended 3 = 12, got {ID:1 Explains:12 CapExplains:8 EndedIDs:[x1 x2 x3 x4]}
== MUTATION: the ended ids are also spent from the cap
RED on the mutant, GREEN on the real file
--- FAIL: TestD84_WindowCredits_TheEndedAllowanceIsBesideTheCap (0.04s)
d84_ended_test.go:93: a: want cap 9 + ended 3 = 12, got {ID:1 Explains:15 CapExplains:12 EndedIDs:[x1 x2 x3]}
2 mutations, 0 not a red-proof
### internal/store/customer_delete.go
== MUTATION: the app record is not in the household's delete
RED on the mutant, GREEN on the real file
--- FAIL: TestD72_AppRecordGoesWithTheHousehold (0.04s)
d84_ended_test.go:152: the residue count does not see the app record: &{Reports:0 AppTelemetry:0 AppLogTails:0 LogTailRequests:0 NotificationPrefs:0 SelfBindTokens:0 ApplianceRegistrations:0 AppLogIssueRefs:0 OffsiteKeys:0 Offsite
1 mutations, 0 not a red-proof
### internal/store/orphan_rows.go
== MUTATION: the app record is not in the orphan sweep
RED on the mutant, GREEN on the real file
--- FAIL: TestD72_AppRecordGoesWithTheHousehold (0.05s)
d84_ended_test.go:165: orphan sweep: map[] <nil>
1 mutations, 0 not a red-proof
### internal/store/offsite_ended.go — the two mutations done again
== MUTATION: REDO (the first mutant did not compile): an absent app's date moves at every report
RED on the mutant, GREEN on the real file
--- FAIL: TestD72_NoteAppList_TheHubsOwnClock (0.05s)
d84_ended_test.go:35: vikunja must be absent since the first report without it (2026-11-01 04:00:00 +0000 UTC), got {Present:false AbsentSince:2026-11-01 08:00:00 +0000 UTC}
== MUTATION: REDO (the first run stayed green: the test had no listed app that was new to the hub; the case was added): a claim makes a LISTED app absent
RED on the mutant, GREEN on the real file
--- FAIL: TestD72_NoteAppList_TheHubsOwnClock (0.04s)
d84_ended_test.go:62: a listed app named in an end record must be present: {Present:false AbsentSince:2026-11-04 22:00:00 +0000 UTC} ok=true
2 mutations, 0 not a red-proof
+32
View File
@@ -1,3 +1,35 @@
## Unreleased (after v0.182.0)
- **The off-site copies of a removed app end after 30 days — the hub's half (R-1374 part 1; D72, D84; `09` §3
decisions 292 and 309; `07-backup-architecture.md` §8.2f). Built, NOT deployed; no box has run it.**
(1) **The hub's own app record.** `store.NoteAppList` (new table `offsite_app_presence`, fed by every controller
report's `stacks.deployed`) keeps, by the hub's clock, when an app left a box's app list; an install forgets it. An
app the hub never saw gets its clock when the box first names it in a pending end record (`offsite.ended_apps`) —
never at a date the box gives. The rows go with the household (`PurgeCustomerResidue`, `PurgeOrphanRows`).
(2) **The hub agrees first** (`offsitekeys.agreeEnded`). `POST /offsite/window-open` now takes `ended` (the box's
due end records with the snapshot ids it would remove) and answers `ended_allowed` / `ended_refused`. An app is
agreed only when the hub's record shows it absent for 30 days; an id only when the hub's own listing of the
repository holds it, uploaded before that absence; an id listed under two apps only when both are agreed. `_shares`
and the marker tag are never eligible. The window's own row holds the agreement and each refusal with its reason,
and the customer page's window block shows them. A clear disagreement (an installed app, or an absence shorter
than 29 days) raises `offsite_ended_app_refused` (warning, operator-only, once per app).
(3) **The allowance is beside the cap (D84).** `max_remove` and `MaxRemove` are unchanged. The close takes
`removed_for_ended_apps` and keeps only ids the hub agreed to; the drop check allows the cap plus those;
`store.WindowCreditsBetween` gives the cap's part (`CapExplains`) and the agreed ids (`EndedIDs`) apart. One
`offsite_ended_app_removed` record per app (info, operator-only, not mailed).
(4) **The 07:10 audit** explains the ended apps' vanished files by name and raises nothing for them; every other
vanished file is judged against the cap's part alone, as before.
(5) **The household's mail (D84).** New box event `offsite_end_scheduled`, a household RECEIPT
(`notify.householdReceiptEvents`): mailed to the household's own notification address once per press, whatever
its group switches say, in its language, with no raw details; the operator is not mailed. A household with no
address gets no mail. `mail.event.offsite_end_scheduled` in both bundles.
**Not built here:** the bytes in the per-app record (the hub cannot see sizes). **A limit, said plainly:** the hub
cannot read a snapshot's tags, so a box that is taken over and has one honestly ended app can name other files
older than that app's end — bounded to files older than 30 days, not closed.
Tests `TestD72_*`, `TestD84_*` (store, offsitekeys, notify, api, web); 44 mutations seen red, 5 of them after a
redo (3 did not compile, 1 was equal to the real code, 1 showed a missing test case)
(`documentation/audits/build-release-4-2026-11-15/A/redproof-hub-*.txt`).
## v0.182.0 (2026-11-15) — release 3 „the hub tells you”: eight silent problems reach the operator; the kernel clean-up and the profile refresh are shown; an off-site account's record outlives an untick (D71); a deleted box or household leaves no rows (D73) (image `sha256:4f8bf649ef85db56a727e77327f5ba7ebcceb739a453c81c5423517882ae4681` built from `35c5b7ce`, CI run 2673) — DEPLOYED 2026-11-15 09:48 CET on the operator's yes (manifest `3fdfab09`, CI run 2674); the first sweep deleted 26 rows
**Read before the deploy:** the daily prune now runs `store.PurgeOrphanRows` — at its first run it deletes the
+92
View File
@@ -0,0 +1,92 @@
package api
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// D72 / D84 (`07-backup-architecture.md` §8.2f): the wire. RED-PROOFS:
// documentation/audits/build-release-4-2026-11-15/A/redproof-hub-api.txt.
// A controller report feeds the hub's OWN app record: the app list, and the apps named in pending end records. The
// box's dates in `ended_apps` never become the hub's date.
func TestD72_ReportFeedsTheHubsOwnAppRecord(t *testing.T) {
h, st, _ := newTestHandler(t)
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ckey", RetrievalPassword: "pp"})
post := func(body string) {
t.Helper()
req := httptest.NewRequest(http.MethodPost, "/api/v1/report", strings.NewReader(body))
req.Header.Set("Authorization", "Bearer ckey")
rr := httptest.NewRecorder()
h.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("report → %d (%s)", rr.Code, rr.Body.String())
}
}
post(`{"customer_id":"c1","stacks":{"deployed":["vikunja","immich"],"available":[]}}`)
before := time.Now().Add(-time.Minute)
// vikunja is gone; the box also holds an end record for an app the hub never saw, dated two years ago.
post(`{"customer_id":"c1","stacks":{"deployed":["immich"],"available":[]},"offsite":{"enabled":true,"ended_apps":[{"app":"oldapp","removed_at":"2024-01-01T00:00:00Z","ends_at":"2024-01-31T00:00:00Z"}]}}`)
m, err := st.AppPresences("c1")
if err != nil {
t.Fatal(err)
}
if !m["immich"].Present {
t.Fatalf("immich is listed: %+v", m)
}
for _, app := range []string{"vikunja", "oldapp"} {
p, ok := m[app]
if !ok || p.Present || p.AbsentSince.Before(before) || p.AbsentSince.After(time.Now().Add(time.Minute)) {
t.Fatalf("%s must be absent since NOW by the hub's clock (the box's 2024 date is not read): %+v ok=%v", app, p, ok)
}
}
// A report with no `stacks` object is not an empty app list: nothing becomes absent.
post(`{"customer_id":"c1"}`)
if m2, _ := st.AppPresences("c1"); !m2["immich"].Present {
t.Fatalf("a report without an app list made immich absent: %+v", m2)
}
}
// The window-open request carries the box's due end records to the service, whole.
func TestD84_WindowOpenCarriesTheEndedClaims(t *testing.T) {
h, st, _ := newTestHandler(t)
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ckey", RetrievalPassword: "pp"})
f := &fakeKeySvc{}
h.SetOffsiteKeyService(f)
id := strings.Repeat("ab", 32)
req := httptest.NewRequest(http.MethodPost, "/api/v1/offsite/window-open/c1",
strings.NewReader(`{"count_before":20,"ended":[{"app":"vikunja","removed_at":"2026-10-01T00:00:00Z","ends_at":"2026-10-31T00:00:00Z","snapshot_ids":["`+id+`"]}]}`))
req.Header.Set("Authorization", "Bearer ckey")
rr := httptest.NewRecorder()
h.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("window-open → %d (%s)", rr.Code, rr.Body.String())
}
if len(f.gotEnded) != 1 || f.gotEnded[0].App != "vikunja" || len(f.gotEnded[0].SnapshotIDs) != 1 || f.gotEnded[0].SnapshotIDs[0] != id {
t.Fatalf("the service did not receive the claim: %+v", f.gotEnded)
}
}
// The household's receipt is a type a box may send; the two hub-made records are not — no box can mint one.
func TestD84_EventTypes(t *testing.T) {
if !allowedEventTypes["offsite_end_scheduled"] {
t.Fatal("offsite_end_scheduled must be accepted from a box (POST /event would answer 400)")
}
if notify.IsOperatorOnly("offsite_end_scheduled") {
t.Fatal("the household's receipt must not be operator-only")
}
for _, et := range []string{"offsite_ended_app_removed", "offsite_ended_app_refused"} {
if allowedEventTypes[et] {
t.Fatalf("%s is hub-made: a box must not be able to POST it", et)
}
if !notify.IsOperatorOnly(et) {
t.Fatalf("%s must be operator-only", et)
}
}
}
+40
View File
@@ -481,6 +481,9 @@ func (h *Handler) handleReport(w http.ResponseWriter, r *http.Request) {
return
}
// D72 (`07` §8.2f part 4): the hub's own record of which apps this box runs, by the hub's clock.
h.noteAppList(payload.CustomerID, body)
// Parse and save app telemetry (backward-compatible — old controllers won't have this field)
var telemetryPayload struct {
AppTelemetry []store.AppTelemetryRecord `json:"app_telemetry"`
@@ -2490,6 +2493,10 @@ var allowedEventTypes = map[string]bool{
// — who, app, the restore's date, size. Operator-only (notify.operatorOnlyEvents) — registered in BOTH in the same
// commit.
"restore_copy_deleted": true,
// D84 (`09` §3 decision 309; `07` §8.2f): the household pressed „delete backup data” for an app — its off-site
// copies end 30 days later. A household RECEIPT (notify.householdReceiptEvents): mailed to the household's own
// address once per press, never to the operator; `mail.event.offsite_end_scheduled` in both bundles.
"offsite_end_scheduled": true,
// Special
"test": true,
}
@@ -3663,3 +3670,36 @@ func (h *Handler) recordHeldApp(customerID, eventType string, raw json.RawMessag
}
h.logger.Printf("[DEBUG] [held-apps] %s/%s held (%s since %s)", customerID, d.StackName, d.Reason, d.Since)
}
// noteAppList keeps the hub's own record of which apps a box runs (store.NoteAppList) from one controller report:
// `stacks.deployed` is the app list, `offsite.ended_apps` the apps the box holds a pending off-site end record for.
// A report with no `stacks` object changes nothing — an absent list is not an empty list. The box's dates in
// `ended_apps` are not read: the clock is the hub's (D72, `09` §3 decision 292).
func (h *Handler) noteAppList(customerID string, body []byte) {
var p struct {
Stacks *struct {
Deployed []string `json:"deployed"`
} `json:"stacks"`
Offsite *struct {
EndedApps []struct {
App string `json:"app"`
// The box's own dates. Carried on the wire for the operator's reading of a stored report; the
// hub decides by its own clock and never by these.
RemovedAt string `json:"removed_at"`
EndsAt string `json:"ends_at"`
} `json:"ended_apps"`
} `json:"offsite"`
}
if err := json.Unmarshal(body, &p); err != nil || p.Stacks == nil {
return
}
var claimed []string
if p.Offsite != nil {
for _, e := range p.Offsite.EndedApps {
claimed = append(claimed, e.App)
}
}
if err := h.store.NoteAppList(customerID, p.Stacks.Deployed, claimed, time.Now()); err != nil {
h.logger.Printf("[WARN] [api] report from %s: the hub's own app record could not be updated: %v", customerID, err)
}
}
+9 -4
View File
@@ -16,7 +16,7 @@ type OffsiteKeyService interface {
RegisterKey(ctx context.Context, customerID, pub string) (offsitekeys.InstallResult, error)
ConfirmKey(ctx context.Context, customerID, fp string) (int, error)
MoveAside(ctx context.Context, customerID string) (string, error)
OpenWindowFor(ctx context.Context, customerID string, countBefore int) (offsitekeys.WindowGrant, error)
OpenWindowFor(ctx context.Context, customerID string, countBefore int, ended ...offsitekeys.EndedClaim) (offsitekeys.WindowGrant, error)
CloseWindowFor(ctx context.Context, customerID string, r offsitekeys.WindowResult) error
RequestAbandon(ctx context.Context, customerID, path string) (offsitekeys.AbandonStatus, error)
CancelAbandon(customerID, by string) (int, error)
@@ -137,12 +137,17 @@ func (h *Handler) handleOffsiteWindowOpen(w http.ResponseWriter, r *http.Request
}
var req struct {
CountBefore int `json:"count_before"`
// D84: the box's pending end records that are due, each with the snapshot ids it would remove. The hub
// agrees per id from its OWN records (offsitekeys/ended.go); the box's dates decide nothing.
Ended []offsitekeys.EndedClaim `json:"ended"`
}
body, _ := io.ReadAll(io.LimitReader(r.Body, 4<<10))
// 1 MiB: a snapshot id is 64 characters, and a removed app can hold a few hundred snapshots. A body that is cut
// does not parse, and then nothing is claimed — the window is still decided on count_before 0, as before.
body, _ := io.ReadAll(io.LimitReader(r.Body, 1<<20))
_ = json.Unmarshal(body, &req)
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Minute)
defer cancel()
g, err := h.offsiteKeys.OpenWindowFor(ctx, customerID, req.CountBefore)
g, err := h.offsiteKeys.OpenWindowFor(ctx, customerID, req.CountBefore, req.Ended...)
if err != nil {
offsiteKeyErr(w, err)
return
@@ -156,7 +161,7 @@ func (h *Handler) handleOffsiteWindowClose(w http.ResponseWriter, r *http.Reques
return
}
var req offsitekeys.WindowResult
body, _ := io.ReadAll(io.LimitReader(r.Body, 8<<10))
body, _ := io.ReadAll(io.LimitReader(r.Body, 1<<20)) // D84: the result names the ended apps' removed ids
if err := json.Unmarshal(body, &req); err != nil || req.WindowID == 0 {
http.Error(w, "body must carry window_id", http.StatusBadRequest)
return
+6 -4
View File
@@ -52,9 +52,10 @@ func TestConsumePassword_RetiredReturnsNoPassword(t *testing.T) {
}
type fakeKeySvc struct {
gotPub string
gotFP string
err error
gotPub string
gotFP string
gotEnded []offsitekeys.EndedClaim
err error
}
func (f *fakeKeySvc) RegisterKey(_ context.Context, _ string, pub string) (offsitekeys.InstallResult, error) {
@@ -65,7 +66,8 @@ func (f *fakeKeySvc) ConfirmKey(_ context.Context, _ string, fp string) (int, er
f.gotFP = fp
return 1, f.err
}
func (f *fakeKeySvc) OpenWindowFor(context.Context, string, int) (offsitekeys.WindowGrant, error) {
func (f *fakeKeySvc) OpenWindowFor(_ context.Context, _ string, _ int, ended ...offsitekeys.EndedClaim) (offsitekeys.WindowGrant, error) {
f.gotEnded = ended
return offsitekeys.WindowGrant{Granted: false, Reason: "not due"}, f.err
}
func (f *fakeKeySvc) CloseWindowFor(context.Context, string, offsitekeys.WindowResult) error {
+1
View File
@@ -89,6 +89,7 @@
"mail.event.app_update_undone": "%s: the update did not work; the app runs on its previous version",
"mail.event.app_update_held": "%s: the app is stopped and needs a restore",
"mail.event.app_stopped_unhealthy": "%s: the app was stopped because it kept crashing",
"mail.event.offsite_end_scheduled": "%s: the off-site copies are marked for deletion",
"mail.event.app_held": "An app is stopped and waits for a person to act",
"mail.event.app_hold_cleared": "The app is no longer held",
"mail.event.app_held_reminder": "An app is still held",
+1
View File
@@ -89,6 +89,7 @@
"mail.event.app_update_undone": "%s: a frissítés nem sikerült, az alkalmazás a korábbi változattal fut",
"mail.event.app_update_held": "%s: az alkalmazás leállítva, visszaállítás szükséges",
"mail.event.app_stopped_unhealthy": "%s: az alkalmazást leállítottuk, mert újra és újra összeomlott",
"mail.event.offsite_end_scheduled": "%s: a távoli másolatok törlésre kijelölve",
"mail.event.app_held": "Egy alkalmazás le van állítva, és emberi beavatkozásra vár",
"mail.event.app_hold_cleared": "Az alkalmazás leállítása megszűnt",
"mail.event.app_held_reminder": "Egy alkalmazás még mindig le van állítva",
+127
View File
@@ -0,0 +1,127 @@
package notify
import (
"strings"
"testing"
)
// D84 (`09` §3 decision 309; `07-backup-architecture.md` §8.2f): the household gets ONE mail at the moment it presses
// „delete backup data”, naming the app, the end date and how to take it back.
// RED-PROOFS: documentation/audits/build-release-4-2026-11-15/A/redproof-hub-mail.txt.
type receiptMail struct{ to, subject, body string }
func receiptDispatcher(t *testing.T, prefsEmail string, enabled []string) (*Dispatcher, *[]receiptMail) {
t.Helper()
st := opOnlyStore(t)
if prefsEmail != "" || enabled != nil {
if err := st.SaveNotificationPrefs("c1", prefsEmail, enabled, 6); err != nil {
t.Fatal(err)
}
}
d := opOnlyDispatcher(t, st, &sentTo{})
var mails []receiptMail
d.sendEmailFn = func(to, subject, body string, _ map[string]string) error {
mails = append(mails, receiptMail{to, subject, body})
return nil
}
return d, &mails
}
const (
receiptHU = "vikunja: az alkalmazás távoli másolatai törlésre vannak kijelölve. Törlésük napja: 2026. 12. 15. Addig a Mentések oldalon a „Mégis megtartom” gombbal visszavonhatod."
receiptEN = "vikunja: the off-site copies of this app are marked for deletion. They are deleted on: 2026-12-15. Until then you can take this back on the Backups page with the \"Keep it after all\" button."
receiptDetails = `{"stack_name":"vikunja","removed_at":"2026-11-15T10:00:00Z","ends_at":"2026-12-15T10:00:00Z","subject":"vikunja|2026-11-15T10:00:00Z"}`
)
// The mail reaches the household's own address although EVERY group switch is off, names the app in the subject, and
// carries the box's sentence (the day, the way back). The operator is not mailed.
func TestD84_Receipt_ReachesTheHouseholdWhateverItsSwitches(t *testing.T) {
d, mails := receiptDispatcher(t, "customer@example.com", []string{})
d.ProcessBoxEvent("c1", "offsite_end_scheduled", "info", receiptHU, "", receiptDetails, "controller")
if len(*mails) != 1 {
t.Fatalf("mails = %d, want exactly one (to the household): %+v", len(*mails), *mails)
}
m := (*mails)[0]
if m.to != "customer@example.com" {
t.Fatalf("the receipt went to %q, not to the household's own address", m.to)
}
if !strings.Contains(m.subject, "vikunja: a távoli másolatok törlésre kijelölve") {
t.Fatalf("the subject does not name the app: %q", m.subject)
}
for _, want := range []string{"2026. 12. 15.", "Mégis megtartom", "Mentések oldalon"} {
if !strings.Contains(m.body, want) {
t.Fatalf("the body lacks %q:\n%s", want, m.body)
}
}
if strings.Contains(m.body, "removed_at") || strings.Contains(m.body, "{") {
t.Fatalf("the body carries the raw details:\n%s", m.body)
}
}
// An English household reads it in English (the box's own second sentence, R-558).
func TestD84_Receipt_FollowsTheHouseholdsLanguage(t *testing.T) {
d, mails := receiptDispatcher(t, "customer@example.com", nil)
if err := d.store.SaveReport("c1", []byte(`{"customer_id":"c1","language":"en"}`)); err != nil {
t.Fatal(err)
}
d.ProcessBoxEvent("c1", "offsite_end_scheduled", "info", receiptHU, receiptEN, receiptDetails, "controller")
if len(*mails) != 1 {
t.Fatalf("mails = %d: %+v", len(*mails), *mails)
}
m := (*mails)[0]
if !strings.Contains(m.subject, "vikunja: the off-site copies are marked for deletion") || !strings.Contains(m.body, "Keep it after all") || !strings.Contains(m.body, "2026-12-15") {
t.Fatalf("not the English receipt:\n%s\n%s", m.subject, m.body)
}
if strings.Contains(m.body, "Mégis megtartom") {
t.Fatalf("Hungarian in an English household's mail:\n%s", m.body)
}
}
// Once per press: the same press delivered twice is one mail; a second app, or the same app removed again later, is
// its own mail — also inside the household's cooldown.
func TestD84_Receipt_OncePerPress(t *testing.T) {
d, mails := receiptDispatcher(t, "customer@example.com", nil)
send := func(app, removedAt string) {
det := `{"stack_name":"` + app + `","removed_at":"` + removedAt + `","subject":"` + app + `|` + removedAt + `"}`
d.ProcessBoxEvent("c1", "offsite_end_scheduled", "info", "x "+app, "", det, "controller")
}
send("vikunja", "2026-11-15T10:00:00Z")
send("vikunja", "2026-11-15T10:00:00Z") // the same press, delivered again
if len(*mails) != 1 {
t.Fatalf("one press gave %d mails", len(*mails))
}
send("immich", "2026-11-15T10:05:00Z") // another app, five minutes later
send("vikunja", "2026-11-15T11:00:00Z") // installed again, removed again
if len(*mails) != 3 {
t.Fatalf("three presses gave %d mails, want 3: %+v", len(*mails), *mails)
}
}
// A household with no address gets no mail (the page is all it has), and nobody else is mailed in its place.
func TestD84_Receipt_NoAddressNoMail(t *testing.T) {
d, mails := receiptDispatcher(t, "", nil)
d.ProcessBoxEvent("c1", "offsite_end_scheduled", "info", receiptHU, "", receiptDetails, "controller")
if len(*mails) != 0 {
t.Fatalf("a household with no address: %d mail(s) went out: %+v", len(*mails), *mails)
}
}
// The receipt register does not open any other type: an ordinary info event still mails nobody, and a type the
// household switched off still stays off.
func TestD84_Receipt_OpensNothingElse(t *testing.T) {
d, mails := receiptDispatcher(t, "customer@example.com", []string{})
d.ProcessBoxEvent("c1", "app_removed", "info", "Alkalmazás eltávolítva.", "", `{"stack_name":"vikunja"}`, "controller")
d.ProcessBoxEvent("c1", "app_stopped_unhealthy", "error", "x", "", `{"stack_name":"vikunja"}`, "controller")
for _, m := range *mails {
if m.to == "customer@example.com" {
t.Fatalf("the household was mailed a type it switched off / an info event: %+v", m)
}
}
if len(householdReceiptEvents) != 1 || !householdReceiptEvents["offsite_end_scheduled"] {
t.Fatalf("the receipt register changed: %v — every entry bypasses the household's switches and needs its own ruling", householdReceiptEvents)
}
if operatorOnlyEvents["offsite_end_scheduled"] {
t.Fatal("a receipt cannot be operator-only")
}
}
+51 -2
View File
@@ -244,6 +244,14 @@ func (d *Dispatcher) processEvent(customerID, eventType, severity, message, mess
return
}
// D84 (`09` §3 decision 309): a household RECEIPT — the mail that tells the household what it has just pressed.
// Before the severity gate (the event is `info`: nothing is wrong) and with no operator mail: the event row is
// the operator's record.
if householdReceiptEvents[eventType] {
d.processCustomer(customerID, eventType, severity, message, messageCustomer, detailsJSON, source)
return
}
// R-182: record-only types are written down and never mailed. Placed BEFORE the severity gate
// so the row is written whatever the severity — the record must not inherit the notification's
// conditions, which is the coupling this whole finding is about.
@@ -632,6 +640,8 @@ var perSubjectCooldownEvents = map[string]bool{
"operator_token_ends_soon": true, "web_cert_ends_soon": true, "hub_daily_job_skipped": true,
"signed_job_not_taken": true, "signed_job_refused": true, "guest_power_gave_up": true, "guest_power_ok": true,
"proxmox_trust_problem": true,
// D72/D84: one refusal per app (details `subject` = the app) — two apps refused in one window are two mails.
"offsite_ended_app_refused": true,
}
// cooldownSubjectSuffix returns ":"+subject for an allow-listed type whose details carry a non-empty `subject`.
@@ -758,6 +768,39 @@ var recordOnlyEvents = map[string]bool{
"recovery_unit_capture_failed": true,
}
// householdReceiptEvents (D84, `09` §3 decision 309) are mails that confirm the household's OWN press of a button
// whose effect comes later and cannot be undone after it. One type today: `offsite_end_scheduled` — the household
// pressed „delete backup data” (or „delete in 30 days”), and the mail names the app, the day the off-site copies are
// deleted and how to take it back. The reviewer's reason, accepted by the operator: the page alone does not warn a
// household whose login someone else uses.
//
// THREE THINGS FOLLOW FROM THAT REASON, each pinned by TestD84_Receipt*:
// - it goes to the household's own notification address by the customer leg (processCustomer) — no second path;
// - it does NOT read the household's group switches (`enabled_events`): whoever holds the login could switch a
// group off first, and this is not a detector the household tunes — it is a receipt;
// - its cooldown is per PRESS (details `subject` = app + the record's own time), so two apps removed in one hour
// are two mails, and one press delivered twice is one mail.
//
// A household with NO address gets no mail — the page is all it has (`07` §8.2f). A blocked household gets none.
// The operator is not mailed: the event row is the record.
var householdReceiptEvents = map[string]bool{
"offsite_end_scheduled": true,
}
// receiptSubjectSuffix returns ":"+subject for a receipt whose details carry a non-empty `subject`.
func receiptSubjectSuffix(eventType, detailsJSON string) string {
if !householdReceiptEvents[eventType] {
return ""
}
var d struct {
Subject string `json:"subject"`
}
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || d.Subject == "" {
return ""
}
return ":" + d.Subject
}
// operatorOnlyEvents are event types that must NEVER reach a customer, whatever their preferences say.
//
// R-97c. This register exists because the guarantee it provides was previously ASSERTED IN A COMMENT
@@ -859,6 +902,10 @@ var operatorOnlyEvents = map[string]bool{
// them. Hub-origin only: deliberately NOT in api.allowedEventTypes, so no box can POST one.
"offsite_snapshots_vanished": true,
"offsite_snapshots_unexplained": true,
// D72/D84 (`07` §8.2f): the off-site copies of a removed app are deleted (info, per app), and a box that claimed
// an end the hub's own records do not show (warning). Hub-origin only: NOT in api.allowedEventTypes.
"offsite_ended_app_removed": true,
"offsite_ended_app_refused": true,
// OS updates (hub v0.130.0, `11` §8 step 2): run failures, rings, switches and approvals are operator facts.
// os_update_applied is deliberately NOT here — it is the household's one line (info: recorded, never mailed).
"os_update_failed": true,
@@ -1092,8 +1139,9 @@ func (d *Dispatcher) processCustomer(customerID, eventType, severity, message, m
return
}
// Check if event type is enabled
if !isEventEnabled(prefs.EnabledEvents, eventType) {
// Check if event type is enabled. A receipt (D84) is not a detector the household switches: see
// householdReceiptEvents.
if !householdReceiptEvents[eventType] && !isEventEnabled(prefs.EnabledEvents, eventType) {
return
}
@@ -1124,6 +1172,7 @@ func (d *Dispatcher) processCustomer(customerID, eventType, severity, message, m
if perAppCustomerCooldownEvents[eventType] {
cooldownKey += cooldownStackSuffixFor(detailsJSON)
}
cooldownKey += receiptSubjectSuffix(eventType, detailsJSON) // D84: one mail per press
d.mu.Lock()
if last, ok := d.custCooldowns[cooldownKey]; ok && time.Since(last) < cooldownDur {
d.mu.Unlock()
+13
View File
@@ -94,6 +94,19 @@ func customerMailCases(lang string) []mailCase {
`{"app":"docmost","stack_name":"docmost","from":{"docmost":"docmost:0.95.0"},"to":{"docmost":"docmost:0.96.0"}}`)
},
},
mailCase{
name: "customer_shape_offsite_end_scheduled_receipt",
comment: "D84: the household's receipt — the entry names the app, the box's sentence is the line, no raw details",
render: func() (string, string) {
box := "vikunja: az alkalmazás távoli másolatai törlésre vannak kijelölve. Törlésük napja: 2026. 12. 15. Addig a Mentések oldalon a „Mégis megtartom” gombbal visszavonhatod."
household := ""
if lang == "en" {
household = "vikunja: the off-site copies of this app are marked for deletion. They are deleted on: 2026-12-15. Until then you can take this back on the Backups page with the \"Keep it after all\" button."
}
return FormatCustomerEmail(lang, "demo-fixture", "offsite_end_scheduled", "info", box, household,
`{"stack_name":"vikunja","removed_at":"2026-11-15T10:00:00Z","ends_at":"2026-12-15T10:00:00Z","subject":"vikunja|2026-11-15T10:00:00Z"}`)
},
},
mailCase{
name: "customer_shape_app_update_held_names_the_app",
comment: "v0.120.0: the held entry names the app; the hold sentence is the line",
+5
View File
@@ -223,6 +223,9 @@ func FormatCustomerEmail(lang, customerID, eventType, severity, message, message
// dashboard shows each issue in the household's language; the mail points there. The
// operator's mail (FormatOperatorEmail) keeps the note. Pinned by r79_health_mail_test.go.
body += b.Msg(lang, "mail.customer.line.health_dashboard")
} else if householdReceiptEvents[eventType] {
// D84: a receipt's details are the record's own fields (app, dates, the press's key) — the sentence above
// already says them in words; the raw JSON would add nothing a household can read.
} else if detailsJSON != "" && detailsJSON != "{}" {
body += b.Msgf(lang, "mail.customer.line.note", detailsJSON)
}
@@ -480,6 +483,8 @@ var appNamedMailEvents = map[string]bool{
"app_update_held": true,
// v0.123.0 (`09` §3 decision 28): the box stopped an app that kept crashing / ran out of memory.
"app_stopped_unhealthy": true,
// D84: the receipt's headline names the app; the box's own sentence (the day, and how to take it back) follows.
"offsite_end_scheduled": true,
}
// stackNameOf reads `stack_name` from an event's details, "" when absent or unreadable.
@@ -0,0 +1,18 @@
SUBJECT: [Felhom] Warning: ?: the off-site copies are marked for deletion
---
Dear Customer,
Your Felhom system sent the following notification:
?: the off-site copies are marked for deletion
Details:
- Server: demo-fixture
- Time: 2026-01-15 10:30
- Level: Warning
- Type: offsite_end_scheduled
If you have any questions, contact your operator.
Best regards,
Felhom.eu monitoring
@@ -0,0 +1,20 @@
# D84: the household's receipt — the entry names the app, the box's sentence is the line, no raw details
SUBJECT: [Felhom] Information: vikunja: the off-site copies are marked for deletion
---
Dear Customer,
Your Felhom system sent the following notification:
vikunja: the off-site copies are marked for deletion
Details:
- Server: demo-fixture
- Time: 2026-01-15 10:30
- Level: Information
- Type: offsite_end_scheduled
- Message: vikunja: the off-site copies of this app are marked for deletion. They are deleted on: 2026-12-15. Until then you can take this back on the Backups page with the "Keep it after all" button.
If you have any questions, contact your operator.
Best regards,
Felhom.eu monitoring
@@ -0,0 +1,18 @@
SUBJECT: [Felhom] Figyelmeztetés: ?: a távoli másolatok törlésre kijelölve
---
Kedves Ügyfél!
A Felhom rendszered a következő értesítést küldte:
?: a távoli másolatok törlésre kijelölve
Részletek:
- Szerver: demo-fixture
- Időpont: 2026-01-15 10:30
- Szint: Figyelmeztetés
- Típus: offsite_end_scheduled
Ha kérdésed van, vedd fel a kapcsolatot az üzemeltetővel.
Üdvözlettel,
Felhom.eu monitoring
@@ -0,0 +1,20 @@
# D84: the household's receipt — the entry names the app, the box's sentence is the line, no raw details
SUBJECT: [Felhom] Információ: vikunja: a távoli másolatok törlésre kijelölve
---
Kedves Ügyfél!
A Felhom rendszered a következő értesítést küldte:
vikunja: a távoli másolatok törlésre kijelölve
Részletek:
- Szerver: demo-fixture
- Időpont: 2026-01-15 10:30
- Szint: Információ
- Típus: offsite_end_scheduled
- Üzenet: vikunja: az alkalmazás távoli másolatai törlésre vannak kijelölve. Törlésük napja: 2026. 12. 15. Addig a Mentések oldalon a „Mégis megtartom” gombbal visszavonhatod.
Ha kérdésed van, vedd fel a kapcsolatot az üzemeltetővel.
Üdvözlettel,
Felhom.eu monitoring
+237
View File
@@ -0,0 +1,237 @@
package offsitekeys
import (
"fmt"
"sort"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// D72 / D84 (`09` §3 decisions 292 and 309; `07-backup-architecture.md` §8.2f) — THE HUB AGREES FIRST.
//
// A box asks, at a window's open, to remove the off-site snapshots of apps the household removed with „delete backup
// data” 30 days ago. Those removals have their own allowance, outside the window's cap — and ONLY for snapshot ids
// this file agreed to. The agreement uses nothing the box says about time:
//
// 1. THE APP. The hub's own record (store.AppPresences, written from the box's reports by the hub's clock) must say
// the app has been absent from that box's app list for EndedAppAfter. An app the newest report lists, an app the
// hub has no record of, an app absent for a shorter time and a reserved name are refused, each with its reason.
// 2. THE ID. A snapshot id is agreed only when the hub's own listing of the repository (the 07:10 audit's stored
// baseline — names and upload times, read over the hub's own login) holds it AND shows it uploaded no later than
// the moment the hub first saw the app absent. An app that is gone makes no new snapshots, so a younger file can
// not be its copy. An id the box lists under two apps is agreed only when both apps are.
//
// WHAT THIS DOES NOT PROVE, said plainly: the hub cannot read a snapshot's tags (the repository is encrypted and the
// password never leaves the box — decision 68). A box that is taken over AND has one honestly ended app can name, under
// that app, other snapshots that are older than the app's end. Rule 2 bounds that to files older than 30 days; it does
// not close it. The same box could already remove half the store per week inside the cap.
// EndedAppAfter is how long the hub must have seen an app absent before it agrees (D72: the number of a restore copy).
const EndedAppAfter = 30 * 24 * time.Hour
const (
EventEndedAppRemoved = "offsite_ended_app_removed" // info: the off-site copies of a removed app are gone (per app)
EventEndedAppRefused = "offsite_ended_app_refused" // warning: a box claimed an end the hub's records do not show
maxEndedClaims = 50 // apps in one request
maxEndedClaimIDs = 5000 // snapshot ids in one request, all apps together
)
// EndedClaim is one pending end record as the box sends it at a window's open. RemovedAt and EndsAt are the BOX's
// dates: they are stored for the operator to read and are never used to decide.
type EndedClaim struct {
App string `json:"app"`
RemovedAt string `json:"removed_at,omitempty"`
EndsAt string `json:"ends_at,omitempty"`
SnapshotIDs []string `json:"snapshot_ids"`
}
// reservedEndedName: the shares pseudo-app, the tier's marker tag and anything shaped like them are never apps a
// household removes (`07` §8.2f, the edge-case table).
func reservedEndedName(app string) bool {
return app == "" || strings.HasPrefix(app, "_") || app == "felhom-offbox" || strings.ContainsAny(app, "/\\ \t\n")
}
// agreeEnded is the PURE decision. presence is the hub's own record; listing the hub's own stored listing (ok=false
// when there is none). It returns the agreed ids (sorted), the agreed ids per agreed app, and every refusal.
func agreeEnded(now time.Time, presence map[string]store.AppPresence, listing store.OffsiteSnapshotListing, haveListing bool, claims []EndedClaim) (ids []string, byApp map[string][]string, refused []store.EndedRefusal) {
if len(claims) > maxEndedClaims {
for _, c := range claims[maxEndedClaims:] {
refused = append(refused, store.EndedRefusal{App: c.App, Reason: fmt.Sprintf("more than %d apps in one request", maxEndedClaims)})
}
claims = claims[:maxEndedClaims]
}
agreedApp := map[string]time.Time{} // app → the hub's absent-since
refusedApp := map[string]bool{}
seen := map[string]bool{}
for _, c := range claims {
if seen[c.App] {
continue
}
seen[c.App] = true
reason := ""
p, known := presence[c.App]
switch {
case reservedEndedName(c.App):
reason = "not an app a household removes"
case !known:
reason = "the hub has no record of this app on this box"
case p.Present:
reason = "the box's newest report lists the app as installed"
case p.AbsentSince.IsZero():
reason = "the hub's record of when the app left cannot be read"
case now.Sub(p.AbsentSince) < EndedAppAfter:
reason = fmt.Sprintf("the hub has seen the app absent since %s — %d full day(s), %d needed",
p.AbsentSince.UTC().Format("2006-01-02 15:04 UTC"), int(now.Sub(p.AbsentSince).Hours()/24), int(EndedAppAfter.Hours()/24))
}
if reason != "" {
refusedApp[c.App] = true
refused = append(refused, store.EndedRefusal{App: c.App, Reason: reason})
continue
}
agreedApp[c.App] = p.AbsentSince
}
// The ids. An id is judged against EVERY app the box listed it under.
type idState struct {
latest time.Time // the earliest absent-since among its apps: the file must be older than every one
refused bool
}
states := map[string]*idState{}
var order []string
total := 0
for _, c := range claims {
for _, id := range c.SnapshotIDs {
total++
if total > maxEndedClaimIDs {
break
}
st, ok := states[id]
if !ok {
st = &idState{}
states[id] = st
order = append(order, id)
}
since, agreed := agreedApp[c.App]
if !agreed {
st.refused = true // listed under an app the hub refused: a living app's copy is never removed
continue
}
if st.latest.IsZero() || since.Before(st.latest) {
st.latest = since
}
}
}
byApp = map[string][]string{}
idRefusals := map[string]int{}
for _, id := range order {
st := states[id]
switch {
case st.refused:
idRefusals["listed under an app the hub refused"]++
continue
case !haveListing:
idRefusals["the hub has no listing of its own for this repository yet"]++
continue
}
at, listed := listing.Entries[id]
switch {
case !snapshotID.MatchString(id) || !listed:
idRefusals["not in the hub's own listing"]++
case at.IsZero():
idRefusals["no upload time in the hub's own listing"]++
case at.After(st.latest.Add(snapTimeSlack)):
idRefusals["uploaded after the hub saw the app leave"]++
default:
ids = append(ids, id)
for _, c := range claims {
if _, ok := agreedApp[c.App]; ok {
for _, cid := range c.SnapshotIDs {
if cid == id {
byApp[c.App] = append(byApp[c.App], id)
}
}
}
}
}
}
if total > maxEndedClaimIDs {
idRefusals[fmt.Sprintf("more than %d snapshot ids in one request", maxEndedClaimIDs)] += total - maxEndedClaimIDs
}
var reasons []string
for r := range idRefusals {
reasons = append(reasons, r)
}
sort.Strings(reasons)
for _, r := range reasons {
refused = append(refused, store.EndedRefusal{App: "", Reason: fmt.Sprintf("%d snapshot id(s) refused: %s", idRefusals[r], r)})
}
sort.Strings(ids)
return ids, byApp, refused
}
// clearDisagreement: a refusal the operator must hear about. A box that asks a little early (its clock is minutes or
// hours ahead of the hub's first sight of the absence) is refused quietly and asks again next week; a box that claims
// an end for an app it reports as installed, or that the hub has seen absent for less than 29 days, is not a matter of
// minutes — its clock is wrong by a day or more (R-1384), or it is not telling the truth.
func clearDisagreement(now time.Time, presence map[string]store.AppPresence, r store.EndedRefusal) bool {
if r.App == "" {
return false
}
p, known := presence[r.App]
// An installed app has no absence date (store.AppPresences), so the zero date covers it too.
if !known || p.AbsentSince.IsZero() {
return true
}
return now.Sub(p.AbsentSince) < EndedAppAfter-24*time.Hour
}
// agreeEndedFor runs the decision for one household against the store, and tells the operator about a clear
// disagreement. It never fails a window: with a store that cannot answer, nothing is agreed.
func (s *Service) agreeEndedFor(customerID string, claims []EndedClaim) (ids []string, byApp map[string][]string, refused []store.EndedRefusal) {
if len(claims) == 0 {
return nil, nil, nil
}
now := s.now()
presence, err := s.Store.AppPresences(customerID)
if err != nil {
s.logf("[WARN] offsitekeys: ended apps %s: the hub's own app record could not be read (%v) — nothing agreed", customerID, err)
for _, c := range claims {
refused = append(refused, store.EndedRefusal{App: c.App, Reason: "the hub's own app record could not be read"})
}
return nil, nil, refused
}
listing, have := s.Store.GetOffsiteSnapshotListing(customerID)
ids, byApp, refused = agreeEnded(now, presence, listing, have, claims)
s.logf("[INFO] offsitekeys: ended apps %s: %d app(s) claimed, %d app(s) with agreed snapshots, %d snapshot id(s) agreed, %d refusal(s)",
customerID, len(claims), len(byApp), len(ids), len(refused))
for _, r := range refused {
s.logf("[WARN] offsitekeys: ended apps %s: REFUSED %q: %s", customerID, r.App, r.Reason)
if clearDisagreement(now, presence, r) {
s.event(customerID, EventEndedAppRefused, "warning",
fmt.Sprintf("Off-site clean-up: the box of %s asked to remove the off-site copies of the removed app %q, and the hub's own records do not agree: %s. Nothing of that app was removed. Check the box's clock and its app list.",
customerID, r.App, r.Reason),
map[string]any{"subject": r.App, "app": r.App, "reason": r.Reason})
}
}
return ids, byApp, refused
}
// endedRemovedByApp groups the removed ids by the app the box listed them under, for the per-app operator record.
func endedRemovedByApp(claims map[string][]string, removed []string) map[string]int {
gone := make(map[string]bool, len(removed))
for _, id := range removed {
gone[id] = true
}
out := map[string]int{}
for app, ids := range claims {
for _, id := range ids {
if gone[id] {
out[app]++
}
}
}
return out
}
+330
View File
@@ -0,0 +1,330 @@
package offsitekeys
import (
"context"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// D72 / D84 (`09` §3 decisions 292, 309; `07-backup-architecture.md` §8.2f): the hub agrees first, by its own clock,
// and the ended apps' allowance covers only ids it agreed to.
// RED-PROOFS: documentation/audits/build-release-4-2026-11-15/A/redproof-hub-ended.txt.
func eid(n int) string { return snapID(900 + n) }
func endedRefusedFor(refused []store.EndedRefusal, app string) string {
for _, r := range refused {
if r.App == app {
return r.Reason
}
}
return ""
}
// The pure decision, one wrong case per row. now is the hub's clock; the claim's own dates are the BOX's and every
// row gives them a value that would agree if the hub read them.
func TestD84_AgreeEnded_TheHubsOwnRecordsDecide(t *testing.T) {
now := time.Date(2026, 12, 20, 3, 0, 0, 0, time.UTC)
days := func(n int) time.Time { return now.Add(-time.Duration(n) * 24 * time.Hour) }
old := days(60) // an upload time well before every absence below
boxSaysLongAgo := func(app string, ids ...string) EndedClaim {
return EndedClaim{App: app, RemovedAt: days(400).Format(time.RFC3339), EndsAt: days(370).Format(time.RFC3339), SnapshotIDs: ids}
}
listing := store.OffsiteSnapshotListing{ListedAt: now.Add(-20 * time.Hour), Entries: map[string]time.Time{
eid(1): old, eid(2): old, eid(3): old, eid(4): days(10), eid(5): {}, eid(6): old, eid(7): days(37),
}}
cases := []struct {
name string
presence map[string]store.AppPresence
listing bool
claims []EndedClaim
wantIDs []string
refused map[string]string // app → a fragment of its reason
}{
{name: "absent 31 days by the hub's clock: agreed",
presence: map[string]store.AppPresence{"vikunja": {AbsentSince: days(31)}}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("vikunja", eid(1), eid(2))}, wantIDs: []string{eid(1), eid(2)}},
{name: "a record with a date in the past, written by a box the hub does not agree with (absent 1 day)",
presence: map[string]store.AppPresence{"vikunja": {AbsentSince: days(1)}}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("vikunja", eid(1))}, refused: map[string]string{"vikunja": "30 needed"}},
{name: "the box's clock is a month ahead: it claims on day 2",
presence: map[string]store.AppPresence{"vikunja": {AbsentSince: days(2)}}, listing: true,
claims: []EndedClaim{{App: "vikunja", RemovedAt: days(2).Format(time.RFC3339), EndsAt: now.Add(-time.Hour).Format(time.RFC3339), SnapshotIDs: []string{eid(1)}}},
refused: map[string]string{"vikunja": "2 full day(s)"}},
{name: "29 days and 23 hours is not 30 days",
presence: map[string]store.AppPresence{"vikunja": {AbsentSince: now.Add(-EndedAppAfter + time.Hour)}}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("vikunja", eid(1))}, refused: map[string]string{"vikunja": "30 needed"}},
{name: "installed again the day before the window: the newest report lists it",
presence: map[string]store.AppPresence{"vikunja": {Present: true}}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("vikunja", eid(1))}, refused: map[string]string{"vikunja": "lists the app as installed"}},
{name: "an app the hub has no record of",
presence: map[string]store.AppPresence{}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("ghost", eid(1))}, refused: map[string]string{"ghost": "no record"}},
{name: "an absence whose date cannot be read is never long ago",
presence: map[string]store.AppPresence{"vikunja": {}}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("vikunja", eid(1))}, refused: map[string]string{"vikunja": "cannot be read"}},
{name: "the shares pseudo-app and the marker tag are never eligible, whatever the record says",
presence: map[string]store.AppPresence{"_shares": {AbsentSince: days(90)}, "felhom-offbox": {AbsentSince: days(90)}}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("_shares", eid(1)), boxSaysLongAgo("felhom-offbox", eid(2))},
refused: map[string]string{"_shares": "not an app", "felhom-offbox": "not an app"}},
{name: "a snapshot with two apps' tags stays until BOTH have ended",
presence: map[string]store.AppPresence{"vikunja": {AbsentSince: days(40)}, "immich": {Present: true}}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("vikunja", eid(1), eid(2)), boxSaysLongAgo("immich", eid(2))},
wantIDs: []string{eid(1)}, refused: map[string]string{"immich": "installed", "": "listed under an app the hub refused"}},
{name: "two ended apps share a snapshot: it goes, and it must be older than BOTH absences",
presence: map[string]store.AppPresence{"vikunja": {AbsentSince: days(40)}, "kimai": {AbsentSince: days(35)}}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("vikunja", eid(1), eid(3)), boxSaysLongAgo("kimai", eid(3))},
wantIDs: []string{eid(1), eid(3)}},
{name: "two ended apps share a snapshot uploaded between their two ends: it is not older than both, so it stays",
presence: map[string]store.AppPresence{"vikunja": {AbsentSince: days(40)}, "kimai": {AbsentSince: days(35)}}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("kimai", eid(7), eid(1)), boxSaysLongAgo("vikunja", eid(7))},
wantIDs: []string{eid(1)}, refused: map[string]string{"": "uploaded after"}},
{name: "an id the hub's own listing does not hold",
presence: map[string]store.AppPresence{"vikunja": {AbsentSince: days(40)}}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("vikunja", eid(1), eid(99), "not-a-snapshot-id")},
wantIDs: []string{eid(1)}, refused: map[string]string{"": "not in the hub's own listing"}},
{name: "a file uploaded AFTER the hub saw the app leave is not that app's copy",
presence: map[string]store.AppPresence{"vikunja": {AbsentSince: days(40)}}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("vikunja", eid(1), eid(4))},
wantIDs: []string{eid(1)}, refused: map[string]string{"": "uploaded after"}},
{name: "a file with no upload time in the hub's listing",
presence: map[string]store.AppPresence{"vikunja": {AbsentSince: days(40)}}, listing: true,
claims: []EndedClaim{boxSaysLongAgo("vikunja", eid(5), eid(6))},
wantIDs: []string{eid(6)}, refused: map[string]string{"": "no upload time"}},
{name: "no listing of the hub's own: no id is agreed",
presence: map[string]store.AppPresence{"vikunja": {AbsentSince: days(40)}}, listing: false,
claims: []EndedClaim{boxSaysLongAgo("vikunja", eid(1))}, refused: map[string]string{"": "no listing of its own"}},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
ids, byApp, refused := agreeEnded(now, c.presence, listing, c.listing, c.claims)
if strings.Join(ids, ",") != strings.Join(c.wantIDs, ",") {
t.Fatalf("agreed ids = %v, want %v (refused: %+v)", shortIDs(ids), shortIDs(c.wantIDs), refused)
}
for app, frag := range c.refused {
if got := endedRefusedFor(refused, app); !strings.Contains(got, frag) {
t.Fatalf("refusal for %q = %q, want it to say %q (all: %+v)", app, got, frag, refused)
}
}
if len(c.refused) == 0 && len(refused) != 0 {
t.Fatalf("nothing should be refused: %+v", refused)
}
n := 0
for _, v := range byApp {
n += len(v)
}
if len(ids) == 0 && n != 0 {
t.Fatalf("no id agreed, yet the per-app record holds %v", byApp)
}
})
}
}
// endedFixture: the pinned household of watchFixture with a stored listing of 20 snapshot files (ids eid(0..19),
// uploaded 60 days ago) and an app record: vikunja absent for 31 days, immich installed.
func endedFixture(t *testing.T) (*Service, *[]evt) {
t.Helper()
s, _, events := watchFixture(t)
now := time.Now().UTC()
l := store.OffsiteSnapshotListing{ListedAt: now.Add(-20 * time.Hour), Entries: map[string]time.Time{}}
for i := 0; i < 20; i++ {
l.Entries[eid(i)] = now.Add(-60 * 24 * time.Hour)
}
if err := s.Store.SetOffsiteSnapshotListing("c1", l); err != nil {
t.Fatal(err)
}
if err := s.Store.NoteAppList("c1", []string{"vikunja", "immich"}, nil, now.Add(-90*24*time.Hour)); err != nil {
t.Fatal(err)
}
if err := s.Store.NoteAppList("c1", []string{"immich"}, nil, now.Add(-31*24*time.Hour)); err != nil {
t.Fatal(err)
}
return s, events
}
func openGranted(t *testing.T, s *Service, count int, ended ...EndedClaim) WindowGrant {
t.Helper()
if err := s.Store.GrantOffsiteWindowOnce("c1"); err != nil {
t.Fatal(err)
}
g, err := s.OpenWindowFor(context.Background(), "c1", count, ended...)
if err != nil || !g.Granted {
t.Fatalf("window not granted: %+v %v", g, err)
}
return g
}
// THE ALLOWANCE IS BESIDE THE CAP (D84). 20 snapshots, cap 10. The policy removes 10 (the whole cap) and the window
// also removes 6 snapshots of the ended app: a fall of 16 raises nothing — and the same fall WITHOUT the hub's
// agreement is the drop alarm it always was. The cap itself is not moved by the ended apps.
func TestD84_Window_TheEndedAllowanceIsBesideTheCap(t *testing.T) {
s, events := endedFixture(t)
ctx := context.Background()
ended := []string{eid(0), eid(1), eid(2), eid(3), eid(4), eid(5)}
g := openGranted(t, s, 20, EndedClaim{App: "vikunja", SnapshotIDs: ended}, EndedClaim{App: "immich", SnapshotIDs: []string{eid(10)}})
if g.MaxRemove != MaxRemove(20) {
t.Fatalf("the cap moved: %d, want %d — the ended apps must not touch it", g.MaxRemove, MaxRemove(20))
}
if strings.Join(g.EndedAllowed, ",") != strings.Join(ended, ",") {
t.Fatalf("agreed ids = %v, want the six of vikunja", shortIDs(g.EndedAllowed))
}
if r := endedRefusedFor(g.EndedRefused, "immich"); !strings.Contains(r, "installed") {
t.Fatalf("the window must say that immich was refused, and why: %+v", g.EndedRefused)
}
if w, _ := s.Store.GetOffsiteWindow(g.WindowID); len(w.EndedAgreed) != 6 || len(w.EndedRefused) == 0 {
t.Fatalf("the window's own row does not hold the agreement and the refusal: %+v", w)
}
if err := s.CloseWindowFor(ctx, "c1", WindowResult{WindowID: g.WindowID, CountBefore: 20, CountAfter: 4, Removed: 16, Outcome: "pruned", RemovedForEndedApps: ended}); err != nil {
t.Fatal(err)
}
if d := ofType(*events, EventWindowDrop); len(d) != 0 {
t.Fatalf("a fall of cap + agreed ended ids raised the drop alarm: %+v", d)
}
rm := ofType(*events, EventEndedAppRemoved)
if len(rm) != 1 || rm[0].sev != "info" || !strings.Contains(rm[0].msg, `"vikunja"`) || !strings.Contains(rm[0].msg, "6 snapshot") {
t.Fatalf("want one info record for vikunja with 6 snapshots: %+v", rm)
}
if w, _ := s.Store.GetOffsiteWindow(g.WindowID); len(w.EndedRemoved) != 6 {
t.Fatalf("the closed window does not hold removed_for_ended_apps: %+v", w)
}
// The same fall, and the box names ids the hub never agreed to (immich's, and invented ones): the drop alarm.
*events = nil
g2 := openGranted(t, s, 20, EndedClaim{App: "immich", SnapshotIDs: []string{eid(10), eid(11), eid(12), eid(13), eid(14), eid(15)}})
if len(g2.EndedAllowed) != 0 {
t.Fatalf("ids of an installed app were agreed: %v", shortIDs(g2.EndedAllowed))
}
if err := s.CloseWindowFor(ctx, "c1", WindowResult{WindowID: g2.WindowID, CountBefore: 20, CountAfter: 4, Removed: 16, Outcome: "pruned",
RemovedForEndedApps: []string{eid(10), eid(11), eid(12), eid(13), eid(14), eid(15)}}); err != nil {
t.Fatal(err)
}
if d := ofType(*events, EventWindowDrop); len(d) != 1 {
t.Fatalf("a fall beyond the cap with NO agreed id must raise the drop alarm, got %+v", *events)
}
if rm := ofType(*events, EventEndedAppRemoved); len(rm) != 0 {
t.Fatalf("a removal the hub did not agree to was recorded as an ended app's: %+v", rm)
}
}
// A window that is not granted agrees to nothing — the box's claim does not open a window and does not reach the row.
func TestD84_Window_NotGrantedAgreesNothing(t *testing.T) {
s, _ := endedFixture(t)
g, err := s.OpenWindowFor(context.Background(), "c1", 20, EndedClaim{App: "vikunja", SnapshotIDs: []string{eid(0)}})
if err != nil || g.Granted || len(g.EndedAllowed) != 0 {
t.Fatalf("weekly windows are off and there is no grant: %+v %v", g, err)
}
}
// The operator hears a CLEAR disagreement once per app, and nothing for a box that asks an hour early.
func TestD84_Window_ARefusalTellsTheOperatorOnlyWhenItIsClear(t *testing.T) {
s, events := endedFixture(t)
now := time.Now().UTC()
// kimai left 2 days ago (the month-ahead clock); nearly left 29 days 23 hours ago (asks an hour early)
_ = s.Store.NoteAppList("c1", []string{"immich", "kimai", "nearly"}, nil, now.Add(-80*24*time.Hour))
_ = s.Store.NoteAppList("c1", []string{"immich", "kimai"}, nil, now.Add(-EndedAppAfter+time.Hour))
_ = s.Store.NoteAppList("c1", []string{"immich"}, nil, now.Add(-2*24*time.Hour))
g := openGranted(t, s, 20,
EndedClaim{App: "kimai", EndsAt: now.Add(-time.Hour).Format(time.RFC3339), SnapshotIDs: []string{eid(1)}},
EndedClaim{App: "nearly", SnapshotIDs: []string{eid(2)}},
EndedClaim{App: "immich", SnapshotIDs: []string{eid(3)}})
if len(g.EndedAllowed) != 0 {
t.Fatalf("agreed: %v", shortIDs(g.EndedAllowed))
}
ref := ofType(*events, EventEndedAppRefused)
var apps []string
for _, e := range ref {
if e.sev != "warning" {
t.Fatalf("severity %q is not in the hub's vocabulary for this event", e.sev)
}
for _, a := range []string{"kimai", "nearly", "immich"} {
if strings.Contains(e.msg, `"`+a+`"`) {
apps = append(apps, a)
}
}
}
if strings.Join(apps, ",") != "kimai,immich" {
t.Fatalf("the operator must hear kimai (2 days) and immich (installed), and not `nearly` (an hour early); heard %v", apps)
}
for _, typ := range []string{EventEndedAppRefused, EventEndedAppRemoved} {
if !notify.IsOperatorOnly(typ) {
t.Fatalf("%s must be operator-only", typ)
}
}
}
// THE 07:10 AUDIT (must-hold 6): the files a window removed for ended apps are explained by name and raise nothing —
// and a file that vanished for any other reason still raises, alone.
func TestD84_Audit_ExplainsEndedAppsAndStillRaisesForTheRest(t *testing.T) {
s, events := endedFixture(t)
ctx := context.Background()
now := time.Now().UTC()
ended := []string{eid(0), eid(1), eid(2)}
g := openGranted(t, s, 20, EndedClaim{App: "vikunja", SnapshotIDs: ended})
if err := s.CloseWindowFor(ctx, "c1", WindowResult{WindowID: g.WindowID, CountBefore: 20, CountAfter: 17, Removed: 3, Outcome: "pruned", RemovedForEndedApps: ended}); err != nil {
t.Fatal(err)
}
*events = nil
listingWithout := func(gone ...string) SnapshotListing {
l := SnapshotListing{Entries: map[string]time.Time{}}
skip := map[string]bool{}
for _, id := range gone {
skip[id] = true
}
for i := 0; i < 20; i++ {
if !skip[eid(i)] {
l.Entries[eid(i)] = now.Add(-60 * 24 * time.Hour)
}
}
return l
}
// exactly the three agreed ids are gone: nothing is raised (the window's cap part explains 0 here: 20-17-3)
if v, _ := s.CheckSnapshotListing("c1", listingWithout(ended...), now.Add(time.Minute)); v != 0 {
t.Fatalf("the ended apps' files were judged as vanished (%d)", v)
}
if e := ofType(*events, EventSnapshotsVanished); len(e) != 0 {
t.Fatalf("an alarm for files the hub itself agreed to remove: %+v", e)
}
// A second household state: the same window, but ONE more file is gone that no window explains.
s2, events2 := endedFixture(t)
g2 := openGranted(t, s2, 20, EndedClaim{App: "vikunja", SnapshotIDs: ended})
if err := s2.CloseWindowFor(ctx, "c1", WindowResult{WindowID: g2.WindowID, CountBefore: 20, CountAfter: 17, Removed: 3, Outcome: "pruned", RemovedForEndedApps: ended}); err != nil {
t.Fatal(err)
}
*events2 = nil
if v, _ := s2.CheckSnapshotListing("c1", listingWithout(append([]string{eid(15)}, ended...)...), now.Add(time.Minute)); v != 1 {
t.Fatalf("one file vanished outside the agreement: want it judged alone, got %d", v)
}
e := ofType(*events2, EventSnapshotsVanished)
if len(e) != 1 || !strings.Contains(e[0].msg, shortID(eid(15))) || strings.Contains(e[0].msg, shortID(eid(0))) {
t.Fatalf("want one alarm naming only the unexplained file %s: %+v", shortID(eid(15)), e)
}
}
// A box that names an ended app's id it did NOT get agreed cannot have it explained by the audit either.
func TestD84_Audit_AnIdTheHubDidNotAgreeToIsNotExplained(t *testing.T) {
s, events := endedFixture(t)
ctx := context.Background()
now := time.Now().UTC()
g := openGranted(t, s, 20, EndedClaim{App: "immich", SnapshotIDs: []string{eid(7)}})
if err := s.CloseWindowFor(ctx, "c1", WindowResult{WindowID: g.WindowID, CountBefore: 20, CountAfter: 20, Removed: 0, Outcome: "nothing", RemovedForEndedApps: []string{eid(7)}}); err != nil {
t.Fatal(err)
}
*events = nil
l := SnapshotListing{Entries: map[string]time.Time{}}
for i := 0; i < 20; i++ {
if i != 7 {
l.Entries[eid(i)] = now.Add(-60 * 24 * time.Hour)
}
}
if v, _ := s.CheckSnapshotListing("c1", l, now.Add(time.Minute)); v != 1 {
t.Fatalf("the file of an installed app vanished and was not judged (%d)", v)
}
if e := ofType(*events, EventSnapshotsVanished); len(e) != 1 {
t.Fatalf("want the vanished alarm: %+v", *events)
}
}
+51 -4
View File
@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"log"
"sort"
"strconv"
"strings"
"time"
@@ -271,6 +272,10 @@ type WindowGrant struct {
NewestAllowed time.Time `json:"newest_allowed,omitempty"`
MaxRemove int `json:"max_remove,omitempty"`
Reason string `json:"reason,omitempty"`
// D84: the snapshot ids of ended apps the hub agreed to — their own allowance, outside MaxRemove — and what it
// refused. A box removes an ended app's snapshot ONLY when its id is in EndedAllowed.
EndedAllowed []string `json:"ended_allowed,omitempty"`
EndedRefused []store.EndedRefusal `json:"ended_refused,omitempty"`
}
// WindowResult is the box's report when it is done.
@@ -281,6 +286,9 @@ type WindowResult struct {
Removed int `json:"removed"`
Outcome string `json:"outcome"`
Reason string `json:"reason"`
// RemovedForEndedApps names the snapshot ids the box removed under the ended apps' allowance (D84). Only ids the
// hub agreed to at the open are counted; anything else the box names here is dropped.
RemovedForEndedApps []string `json:"removed_for_ended_apps,omitempty"`
}
// MaxRemove is the most snapshots one window may remove: HALF of what was there, at least 5. The ruled
@@ -331,7 +339,7 @@ func (s *Service) GrantLargeWindow(customerID string, maxRemove int) error {
// OpenWindowFor decides and, if due, opens the window: a deleting line for the box's CONFIRMED key is
// prepended (first match wins), and a ledger row bounds it to 20 minutes.
func (s *Service) OpenWindowFor(ctx context.Context, customerID string, countBefore int) (WindowGrant, error) {
func (s *Service) OpenWindowFor(ctx context.Context, customerID string, countBefore int, ended ...EndedClaim) (WindowGrant, error) {
oneShot, raisedMax := s.Store.TakeOffsiteWindowGrant(customerID)
last := s.Store.LastOffsiteWindowOpened(customerID)
due := last.IsZero() || s.now().Sub(last) >= windowCadence
@@ -368,6 +376,16 @@ func (s *Service) OpenWindowFor(ctx context.Context, customerID string, countBef
return WindowGrant{}, err
}
g := WindowGrant{Granted: true, WindowID: id, NewestAllowed: now.UTC(), MaxRemove: maxRemove}
// D84: the ended apps. Decided AFTER the window exists and only for a granted window, so a refused or not-due
// window agrees to nothing; a ledger write that fails agrees to nothing either (the box then removes none).
if agreed, byApp, refused := s.agreeEndedFor(customerID, ended); len(agreed) > 0 || len(refused) > 0 {
if werr := s.Store.SetOffsiteWindowEnded(id, agreed, byApp, refused); werr != nil {
s.logf("[WARN] offsitekeys: window %d: the ended apps' agreement could not be recorded (%v) — nothing agreed", id, werr)
g.EndedRefused = []store.EndedRefusal{{Reason: "the hub could not record its agreement"}}
} else {
g.EndedAllowed, g.EndedRefused = agreed, refused
}
}
s.logf("[WARN] offsitekeys: clean-up window %d OPENED for %s (key %s, %d snapshot(s), max %d removed%s, closes by %s, one-shot=%v)",
id, customerID, k.Fingerprint, countBefore, g.MaxRemove, map[bool]string{true: " — OPERATOR-RAISED cap (default " + strconv.Itoa(MaxRemove(countBefore)) + ")", false: ""}[raised],
now.Add(windowLength).UTC().Format(time.RFC3339), oneShot)
@@ -387,14 +405,43 @@ func (s *Service) CloseWindowFor(ctx context.Context, customerID string, r Windo
if err := s.Reg.CloseWindow(ctx, t, pw); err != nil {
return err // the sweep retries at closes_by
}
// D84: the ended apps' own allowance — only ids the hub agreed to at the open (the store checks each against the
// row). Recorded BEFORE the row closes, so a reader of a closed window always finds what it explains.
var endedRemoved []string
if len(r.RemovedForEndedApps) > 0 {
kept, eerr := s.Store.SetOffsiteWindowEndedRemoved(w.ID, r.RemovedForEndedApps)
if eerr != nil {
s.logf("[WARN] offsitekeys: window %d: the ended apps' removals could not be recorded (%v) — they are judged under the cap", w.ID, eerr)
}
endedRemoved = kept
if len(kept) != len(r.RemovedForEndedApps) {
s.logf("[WARN] offsitekeys: window %d: the box named %d id(s) as removed for ended apps, the hub had agreed to %d of them — the rest count under the cap",
w.ID, len(r.RemovedForEndedApps), len(kept))
}
}
if _, err := s.Store.CloseOffsiteWindowRow(w.ID, r.CountAfter, r.Outcome, "box"); err != nil {
s.logf("[WARN] offsitekeys: ledger close %d: %v", w.ID, err)
}
drop := w.CountBefore - r.CountAfter
allowed := windowCap(w)
s.logf("[INFO] offsitekeys: clean-up window %d CLOSED for %s: outcome=%s, %d -> %d (drop %d, allowed %d)",
w.ID, customerID, r.Outcome, w.CountBefore, r.CountAfter, drop, allowed)
// The cap's own rule is unchanged (windowCap); the ended apps' ids are beside it, never part of it.
allowed := windowCap(w) + len(endedRemoved)
s.logf("[INFO] offsitekeys: clean-up window %d CLOSED for %s: outcome=%s, %d -> %d (drop %d, allowed %d = cap %d + %d for ended apps)",
w.ID, customerID, r.Outcome, w.CountBefore, r.CountAfter, drop, allowed, windowCap(w), len(endedRemoved))
details := map[string]any{"window_id": w.ID, "count_before": w.CountBefore, "count_after": r.CountAfter, "outcome": r.Outcome, "reason": r.Reason}
if len(endedRemoved) > 0 {
details["removed_for_ended_apps"] = len(endedRemoved)
byApp := endedRemovedByApp(w.EndedApps, endedRemoved)
apps := make([]string, 0, len(byApp))
for app := range byApp {
apps = append(apps, app)
}
sort.Strings(apps)
for _, app := range apps {
s.event(customerID, EventEndedAppRemoved, "info",
fmt.Sprintf("Off-site clean-up window %d: the off-site copies of the removed app %q are deleted (%d snapshot(s)), 30 days after the household pressed „delete backup data”.", w.ID, app, byApp[app]),
map[string]any{"subject": app, "app": app, "snapshots": byApp[app], "window_id": w.ID})
}
}
switch {
case drop > allowed:
s.event(customerID, EventWindowDrop, "error",
+25 -1
View File
@@ -119,9 +119,33 @@ func (s *Service) CheckSnapshotListing(customerID string, l SnapshotListing, now
customerID, len(gone), prev.ListedAt.Format(time.RFC3339))
default:
credits, unknown := s.Store.WindowCreditsBetween(customerID, prev.ListedAt.Add(-snapWindowSlack), now)
// D84 (`09` §3 decision 309): the ids a window removed for ENDED APPS are explained BY NAME — only ids the
// hub itself agreed to at that window's open (store.WindowCredit.EndedIDs). They are taken out first and
// raise nothing. Every other vanished file is judged exactly as before, against the cap's part alone.
endedOK := map[string]bool{}
explains := 0
for _, c := range credits {
explains += c.Explains
explains += c.CapExplains
for _, id := range c.EndedIDs {
endedOK[id] = true
}
}
var forEnded []string
rest := gone[:0:0]
for _, id := range gone {
if endedOK[id] {
forEnded = append(forEnded, id)
} else {
rest = append(rest, id)
}
}
if len(forEnded) > 0 {
s.logf("[INFO] offsitekeys: snapshots %s: %d file(s) gone since %s are removed_for_ended_apps — ids the hub agreed to in a clean-up window: %s",
customerID, len(forEnded), prev.ListedAt.Format(time.RFC3339), idList(forEnded))
}
gone = rest
if len(gone) == 0 {
break
}
if unknown || len(gone) > explains {
vanished = len(gone)
+4 -1
View File
@@ -51,6 +51,8 @@ type CustomerResidue struct {
OffsiteKeys int
OffsiteWindows int
OffsiteAbandonRequests int
// D72: the hub's own record of which apps the household's box runs (offsite_ended.go).
OffsiteAppPresence int
}
// Total is the single number the "is there anything left at all?" decision hangs on.
@@ -60,7 +62,7 @@ func (r *CustomerResidue) Total() int {
}
return r.Reports + r.AppTelemetry + r.AppLogTails + r.LogTailRequests +
r.NotificationPrefs + r.SelfBindTokens + r.ApplianceRegistrations + r.AppLogIssueRefs +
r.OffsiteKeys + r.OffsiteWindows + r.OffsiteAbandonRequests
r.OffsiteKeys + r.OffsiteWindows + r.OffsiteAbandonRequests + r.OffsiteAppPresence
}
// residueQueries is the ONE list both the counter and the purge walk, so a table can never be
@@ -79,6 +81,7 @@ var residueQueries = []struct {
{"offsite_keys", func(r *CustomerResidue) *int { return &r.OffsiteKeys }},
{"offsite_windows", func(r *CustomerResidue) *int { return &r.OffsiteWindows }},
{"offsite_abandon_requests", func(r *CustomerResidue) *int { return &r.OffsiteAbandonRequests }},
{"offsite_app_presence", func(r *CustomerResidue) *int { return &r.OffsiteAppPresence }},
}
// CustomerResidue counts the customer's report-derived + binding residue. Read-only.
+172
View File
@@ -0,0 +1,172 @@
package store
import (
"testing"
"time"
)
// D72 (`09` §3 decision 292; `07` §8.2f part 4): the hub keeps ITS OWN record of when an app left a box's app list,
// by the hub's clock. RED-PROOFS: documentation/audits/build-release-4-2026-11-15/A/redproof-hub-store.txt.
func TestD72_NoteAppList_TheHubsOwnClock(t *testing.T) {
s := newTestStore(t)
t0 := time.Date(2026, 11, 1, 3, 0, 0, 0, time.UTC)
note := func(at time.Time, deployed, claimed []string) {
t.Helper()
if err := s.NoteAppList("c1", deployed, claimed, at); err != nil {
t.Fatal(err)
}
}
get := func() map[string]AppPresence {
t.Helper()
m, err := s.AppPresences("c1")
if err != nil {
t.Fatal(err)
}
return m
}
note(t0, []string{"vikunja", "immich"}, nil)
if m := get(); !m["vikunja"].Present || !m["immich"].Present || len(m) != 2 {
t.Fatalf("first report: %+v", m)
}
// vikunja leaves: absent since the FIRST report without it, and a later report does not move that time.
note(t0.Add(time.Hour), []string{"immich"}, nil)
note(t0.Add(5*time.Hour), []string{"immich"}, nil)
if p := get()["vikunja"]; p.Present || !p.AbsentSince.Equal(t0.Add(time.Hour)) {
t.Fatalf("vikunja must be absent since the first report without it (%s), got %+v", t0.Add(time.Hour), p)
}
if p := get()["immich"]; !p.Present || !p.AbsentSince.IsZero() {
t.Fatalf("immich is still listed: %+v", p)
}
// installed again: the absence is forgotten; removed again later: a NEW absence, never the old date.
note(t0.Add(48*time.Hour), []string{"immich", "vikunja"}, nil)
if p := get()["vikunja"]; !p.Present || !p.AbsentSince.IsZero() {
t.Fatalf("an install must end the absence: %+v", p)
}
note(t0.Add(72*time.Hour), []string{"immich"}, nil)
if p := get()["vikunja"]; p.Present || !p.AbsentSince.Equal(t0.Add(72*time.Hour)) {
t.Fatalf("the second removal must start a new absence at its own time: %+v", p)
}
// An app the hub never saw, named in a pending end record: the hub's clock starts when it first hears the claim.
note(t0.Add(80*time.Hour), []string{"immich"}, []string{"oldapp", "immich"})
note(t0.Add(90*time.Hour), []string{"immich"}, []string{"oldapp"})
m := get()
if p, ok := m["oldapp"]; !ok || p.Present || !p.AbsentSince.Equal(t0.Add(80*time.Hour)) {
t.Fatalf("a claimed, never-seen app must be absent since the first claim (%s): %+v ok=%v", t0.Add(80*time.Hour), p, ok)
}
if !m["immich"].Present {
t.Fatalf("a claim must never make a listed app absent: %+v", m["immich"])
}
// An app that is listed AND named in an end record in the very report that first shows it: it is installed.
note(t0.Add(91*time.Hour), []string{"immich", "fresh"}, []string{"oldapp", "fresh"})
if p, ok := get()["fresh"]; !ok || !p.Present {
t.Fatalf("a listed app named in an end record must be present: %+v ok=%v", p, ok)
}
// another household's box is not touched
if o, _ := s.AppPresences("c2"); len(o) != 0 {
t.Fatalf("c2 has rows: %+v", o)
}
}
// D84 (`09` §3 decision 309): a window's credit for ended apps is BESIDE its cap. The cap's part is what it was;
// the ended part is only ids the hub agreed to.
func TestD84_WindowCredits_TheEndedAllowanceIsBesideTheCap(t *testing.T) {
s := newTestStore(t)
at := func(ts string) time.Time { v, _ := time.Parse("2006-01-02 15:04:05", ts); return v }
from, to := at("2026-09-30 00:00:00"), at("2026-10-02 00:00:00")
ins := func(cust, closed string, before, after, maxRemove any, agreed, removed any) {
t.Helper()
var c any
if closed != "" {
c = closed
}
if _, err := s.db.Exec(`INSERT INTO offsite_windows (customer_id, opened_at, closes_by, closed_at, count_before, count_after, max_remove, ended_agreed, ended_removed) VALUES (?, '2026-10-01 03:00:00', '2026-10-01 23:00:00', ?, ?, ?, ?, ?, ?)`,
cust, c, before, after, maxRemove, agreed, removed); err != nil {
t.Fatal(err)
}
}
one := func(cust string) WindowCredit {
t.Helper()
cs, unk := s.WindowCreditsBetween(cust, from, to)
if unk || len(cs) != 1 {
t.Fatalf("%s: want one known credit, got %+v unknown=%v", cust, cs, unk)
}
return cs[0]
}
// closed, 40 -> 28: 3 ids for ended apps (named by the box, agreed by the hub) + 9 under a cap of 20
ins("a", "2026-10-01 03:10:00", 40, 28, 20, `["x1","x2","x3","x4"]`, `["x1","x2","x3"]`)
if c := one("a"); c.CapExplains != 9 || c.Explains != 12 || len(c.EndedIDs) != 3 {
t.Fatalf("a: want cap 9 + ended 3 = 12, got %+v", c)
}
// a lying box: count fell 40 -> 0; the cap still explains only 20, the ended part only the 3 named ids
ins("l", "2026-10-01 03:10:00", 40, 0, 20, `["x1","x2","x3","x4"]`, `["x1","x2","x3"]`)
if c := one("l"); c.CapExplains != 20 || c.Explains != 23 {
t.Fatalf("lying box: want cap 20 + ended 3, got %+v", c)
}
// no box result (still open): the cap, plus every id the hub AGREED to — never more
ins("o", "", 40, nil, 20, `["x1","x2","x3","x4"]`, nil)
if c := one("o"); c.CapExplains != 20 || c.Explains != 24 || len(c.EndedIDs) != 4 {
t.Fatalf("open window: want cap 20 + the 4 agreed ids, got %+v", c)
}
// only ended apps went (the policy removed nothing): the window still explains them
ins("e", "2026-10-01 03:10:00", 40, 38, 20, `["x1","x2"]`, `["x1","x2"]`)
if c := one("e"); c.CapExplains != 0 || c.Explains != 2 || len(c.EndedIDs) != 2 {
t.Fatalf("ended only: want cap 0 + ended 2, got %+v", c)
}
// a window with no ended app is exactly what it was
ins("p", "2026-10-01 03:10:00", 40, 31, 20, nil, nil)
if c := one("p"); c.CapExplains != 9 || c.Explains != 9 || len(c.EndedIDs) != 0 {
t.Fatalf("plain window changed: %+v", c)
}
}
// The box's result may name only ids the hub agreed to at the open: anything else is dropped before it is stored.
func TestD84_EndedRemoved_OnlyAgreedIdsAreStored(t *testing.T) {
s := newTestStore(t)
id, err := s.OpenOffsiteWindowRow("c1", time.Now().Add(20*time.Minute), 40, 20)
if err != nil {
t.Fatal(err)
}
if err := s.SetOffsiteWindowEnded(id, []string{"a1", "a2"}, map[string][]string{"vikunja": {"a1", "a2"}}, []EndedRefusal{{App: "immich", Reason: "listed"}}); err != nil {
t.Fatal(err)
}
kept, err := s.SetOffsiteWindowEndedRemoved(id, []string{"a2", "zz", "a2", "a1", "living"})
if err != nil || len(kept) != 2 || kept[0] != "a1" || kept[1] != "a2" {
t.Fatalf("kept = %v err=%v, want [a1 a2]", kept, err)
}
w, _ := s.GetOffsiteWindow(id)
if len(w.EndedRemoved) != 2 || len(w.EndedAgreed) != 2 || len(w.EndedRefused) != 1 || w.EndedRefused[0].App != "immich" || len(w.EndedApps["vikunja"]) != 2 {
t.Fatalf("stored row: %+v", w)
}
// a window that agreed to nothing stores nothing, whatever the box names
id2, _ := s.OpenOffsiteWindowRow("c1", time.Now().Add(20*time.Minute), 40, 20)
if kept, _ := s.SetOffsiteWindowEndedRemoved(id2, []string{"a1"}); len(kept) != 0 {
t.Fatalf("a window with no agreement stored %v", kept)
}
if w2, _ := s.GetOffsiteWindow(id2); len(w2.EndedRemoved) != 0 {
t.Fatalf("stored on a window with no agreement: %+v", w2)
}
}
// The hub's app record goes with the household (D73's rule for every household-keyed table).
func TestD72_AppRecordGoesWithTheHousehold(t *testing.T) {
s := newTestStore(t)
if err := s.NoteAppList("gone", []string{"vikunja"}, nil, time.Now()); err != nil {
t.Fatal(err)
}
if r, err := s.CustomerResidue("gone"); err != nil || r.OffsiteAppPresence != 1 || r.Total() < 1 {
t.Fatalf("the residue count does not see the app record: %+v %v", r, err)
}
if err := s.PurgeCustomerResidue("gone"); err != nil {
t.Fatal(err)
}
if m, _ := s.AppPresences("gone"); len(m) != 0 {
t.Fatalf("the app record outlived the household: %+v", m)
}
// and the daily orphan sweep removes what an earlier delete left
if err := s.NoteAppList("ghost", []string{"vikunja"}, nil, time.Now()); err != nil {
t.Fatal(err)
}
if n, err := s.PurgeOrphanRows(); err != nil || n["offsite_app_presence"] != 1 {
t.Fatalf("orphan sweep: %+v %v", n, err)
}
}
+217
View File
@@ -0,0 +1,217 @@
package store
import (
"database/sql"
"encoding/json"
"fmt"
"sort"
"time"
)
// D72 / D84 (`09` §3 decisions 292 and 309; `07-backup-architecture.md` §8.2f part 4) — THE HUB'S OWN RECORD of which
// apps a box runs, kept by the hub's clock. A box may ask a clean-up window to remove the off-site snapshots of an app
// the household removed 30 days ago; the hub agrees only when THIS table says the app has been absent from that box's
// reported app list for 30 days. The box's own dates are never read for that.
//
// One row per (household, app) the hub has seen in a report's `stacks.deployed`, or that the box named in a pending
// end record. `absent_since` is the hub's time of the FIRST report that did not list the app; it is cleared the moment
// a report lists the app again (an install ends every claim). Rows go with the household (residueQueries,
// orphanTables).
// migrateOffsiteEnded creates the presence table and the three window columns. Idempotent.
func (s *Store) migrateOffsiteEnded() error {
if _, err := s.db.Exec(`
CREATE TABLE IF NOT EXISTS offsite_app_presence (
customer_id TEXT NOT NULL,
app TEXT NOT NULL,
absent_since TEXT,
PRIMARY KEY (customer_id, app)
);`); err != nil {
return fmt.Errorf("offsite_app_presence: %w", err)
}
// What the hub agreed to at the open, what it refused, and what the box then named as removed (only ids the hub
// agreed to are kept). JSON text; NULL on every window that had no ended app.
s.db.Exec("ALTER TABLE offsite_windows ADD COLUMN ended_agreed TEXT")
s.db.Exec("ALTER TABLE offsite_windows ADD COLUMN ended_refused TEXT")
s.db.Exec("ALTER TABLE offsite_windows ADD COLUMN ended_removed TEXT")
s.db.Exec("ALTER TABLE offsite_windows ADD COLUMN ended_apps TEXT") // agreed app → its agreed ids (the per-app record)
return nil
}
// AppPresence is the hub's record of one app on one box.
type AppPresence struct {
// Present: the newest report lists the app.
Present bool
// AbsentSince is the hub's time of the first report without the app (zero while Present).
AbsentSince time.Time
}
// maxPresenceApps bounds what one report may add: a box's lists are the box's own word.
const maxPresenceApps = 300
// NoteAppList updates the record from ONE report: `deployed` is the report's app list, `claimed` the apps the box
// names in pending off-site end records. now is the HUB's clock.
//
// - an app in `deployed` is present (its absence, if any, is forgotten);
// - an app the hub knew as present and that is not in `deployed` is absent since now;
// - a claimed app the hub has NO row for and that is not in `deployed` gets a row, absent since now — the hub's
// clock for an app removed before the hub kept this record starts when the hub first hears the claim, never at a
// date the box names.
//
// It writes only when something changes, so a steady box costs one read per report.
func (s *Store) NoteAppList(customerID string, deployed, claimed []string, now time.Time) error {
if customerID == "" {
return fmt.Errorf("NoteAppList: empty customer_id")
}
if len(deployed) > maxPresenceApps {
deployed = deployed[:maxPresenceApps]
}
if len(claimed) > maxPresenceApps {
claimed = claimed[:maxPresenceApps]
}
cur, err := s.AppPresences(customerID)
if err != nil {
return err
}
stamp := now.UTC().Format(time.RFC3339)
in := make(map[string]bool, len(deployed))
type change struct {
app string
absent sql.NullString
}
var changes []change
for _, app := range deployed {
if app == "" || in[app] {
continue
}
in[app] = true
if p, ok := cur[app]; !ok || !p.Present {
changes = append(changes, change{app: app})
}
}
for app, p := range cur {
if p.Present && !in[app] {
changes = append(changes, change{app: app, absent: sql.NullString{String: stamp, Valid: true}})
}
}
seen := map[string]bool{}
for _, app := range claimed {
if app == "" || in[app] || seen[app] {
continue
}
seen[app] = true
if _, ok := cur[app]; !ok {
changes = append(changes, change{app: app, absent: sql.NullString{String: stamp, Valid: true}})
}
}
if len(changes) == 0 {
return nil
}
tx, err := s.db.Begin()
if err != nil {
return err
}
defer tx.Rollback()
for _, c := range changes {
if _, err := tx.Exec(`INSERT INTO offsite_app_presence (customer_id, app, absent_since) VALUES (?, ?, ?)
ON CONFLICT(customer_id, app) DO UPDATE SET absent_since = excluded.absent_since`, customerID, c.app, c.absent); err != nil {
return fmt.Errorf("NoteAppList %s/%s: %w", customerID, c.app, err)
}
}
return tx.Commit()
}
// AppPresences returns the hub's record for one household, by app.
func (s *Store) AppPresences(customerID string) (map[string]AppPresence, error) {
rows, err := s.db.Query(`SELECT app, absent_since FROM offsite_app_presence WHERE customer_id = ?`, customerID)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]AppPresence{}
for rows.Next() {
var app string
var absent sql.NullString
if err := rows.Scan(&app, &absent); err != nil {
return nil, err
}
p := AppPresence{Present: !absent.Valid || absent.String == ""}
if !p.Present {
// A date that cannot be read stays the zero time, and the caller REFUSES an absent app with a zero
// AbsentSince: an unreadable date is never "absent long ago".
if t, perr := time.Parse(time.RFC3339, absent.String); perr == nil {
p.AbsentSince = t
}
}
out[app] = p
}
return out, rows.Err()
}
// EndedRefusal is one app the hub refused in a window, with the reason in plain words.
type EndedRefusal struct {
App string `json:"app"`
Reason string `json:"reason"`
}
// SetOffsiteWindowEnded records, on the window's own row, the snapshot ids the hub agreed to for ended apps and the
// apps it refused. Called once, at the open.
func (s *Store) SetOffsiteWindowEnded(id int64, agreed []string, byApp map[string][]string, refused []EndedRefusal) error {
var a, r, m sql.NullString
if len(byApp) > 0 {
b, _ := json.Marshal(byApp)
m = sql.NullString{String: string(b), Valid: true}
}
if len(agreed) > 0 {
b, _ := json.Marshal(agreed)
a = sql.NullString{String: string(b), Valid: true}
}
if len(refused) > 0 {
b, _ := json.Marshal(refused)
r = sql.NullString{String: string(b), Valid: true}
}
_, err := s.db.Exec(`UPDATE offsite_windows SET ended_agreed = ?, ended_refused = ?, ended_apps = ? WHERE id = ?`, a, r, m, id)
return err
}
// SetOffsiteWindowEndedRemoved records which agreed ids the box named as removed. The caller passes only ids the hub
// agreed to; this function checks it again against the row, so an id the hub never agreed to can not be stored.
func (s *Store) SetOffsiteWindowEndedRemoved(id int64, removed []string) ([]string, error) {
w, err := s.GetOffsiteWindow(id)
if err != nil || w == nil {
return nil, fmt.Errorf("SetOffsiteWindowEndedRemoved: window %d: %v", id, err)
}
ok := make(map[string]bool, len(w.EndedAgreed))
for _, a := range w.EndedAgreed {
ok[a] = true
}
var kept []string
dup := map[string]bool{}
for _, r := range removed {
if ok[r] && !dup[r] {
dup[r] = true
kept = append(kept, r)
}
}
sort.Strings(kept)
var v sql.NullString
if len(kept) > 0 {
b, _ := json.Marshal(kept)
v = sql.NullString{String: string(b), Valid: true}
}
if _, err := s.db.Exec(`UPDATE offsite_windows SET ended_removed = ? WHERE id = ?`, v, id); err != nil {
return nil, err
}
return kept, nil
}
func jsonStrings(v sql.NullString) []string {
if !v.Valid || v.String == "" {
return nil
}
var out []string
if json.Unmarshal([]byte(v.String), &out) != nil {
return nil
}
return out
}
+45 -8
View File
@@ -2,6 +2,7 @@ package store
import (
"database/sql"
"encoding/json"
"fmt"
"strconv"
"strings"
@@ -78,6 +79,14 @@ type OffsiteWindow struct {
// MaxRemove is the cap this window was opened with (R-833: an operator grant may raise it for one
// window). 0 on rows written before v0.129.0 — readers fall back to the default cap then.
MaxRemove int
// D84 (`09` §3 decision 309): the off-site snapshots of ended apps have their own allowance, outside MaxRemove.
// EndedAgreed are the snapshot ids the hub agreed to at the open; EndedRemoved the agreed ids the box named as
// removed in its result; EndedRefused the apps the hub refused, each with its reason. Empty on most windows.
EndedAgreed []string
EndedRemoved []string
EndedRefused []EndedRefusal
// EndedApps: each agreed app with its agreed ids (the per-app operator record at the close).
EndedApps map[string][]string
}
// OpenOffsiteWindowRow records a window the hub just opened, with the cap it was opened under.
@@ -107,8 +116,9 @@ func (s *Store) GetOffsiteWindow(id int64) (*OffsiteWindow, error) {
var opened, closesBy string
var closed, boxRes, reason sql.NullString
var before, after, maxRm sql.NullInt64
err := s.db.QueryRow(`SELECT id, customer_id, opened_at, closes_by, closed_at, count_before, count_after, box_result, close_reason, max_remove FROM offsite_windows WHERE id = ?`, id).
Scan(&w.ID, &w.CustomerID, &opened, &closesBy, &closed, &before, &after, &boxRes, &reason, &maxRm)
var endAgreed, endRefused, endRemoved, endApps sql.NullString
err := s.db.QueryRow(`SELECT id, customer_id, opened_at, closes_by, closed_at, count_before, count_after, box_result, close_reason, max_remove, ended_agreed, ended_refused, ended_removed, ended_apps FROM offsite_windows WHERE id = ?`, id).
Scan(&w.ID, &w.CustomerID, &opened, &closesBy, &closed, &before, &after, &boxRes, &reason, &maxRm, &endAgreed, &endRefused, &endRemoved, &endApps)
if err == sql.ErrNoRows {
return nil, nil
}
@@ -122,6 +132,13 @@ func (s *Store) GetOffsiteWindow(id int64) (*OffsiteWindow, error) {
w.CountBefore, w.CountAfter = int(before.Int64), int(after.Int64)
w.BoxResult, w.CloseReason = boxRes.String, reason.String
w.MaxRemove = int(maxRm.Int64)
w.EndedAgreed, w.EndedRemoved = jsonStrings(endAgreed), jsonStrings(endRemoved)
if endRefused.Valid && endRefused.String != "" {
_ = json.Unmarshal([]byte(endRefused.String), &w.EndedRefused)
}
if endApps.Valid && endApps.String != "" {
_ = json.Unmarshal([]byte(endApps.String), &w.EndedApps)
}
return &w, nil
}
@@ -367,8 +384,16 @@ func (s *Store) ForceOffsiteAbandonDueForTest(id int64) error {
// WindowCredit is how many snapshots one clean-up window may explain (R-435).
type WindowCredit struct {
ID int64
ID int64
// Explains is the most snapshots this window may explain in a COUNT: the cap's part plus the ended apps' part.
Explains int
// CapExplains is the cap's part alone (at most max_remove) — what a reader that explains the ended apps' ids
// BY NAME (the 07:10 listing check) may still spend on the other vanished files.
CapExplains int
// EndedIDs are the snapshot ids of ended apps this window explains by name (D84): the agreed ids the box named
// as removed, or — for a window with no box result (still open, closed by timeout) — every id the hub agreed to.
// Never an id the hub did not agree to.
EndedIDs []string
}
// WindowCreditsBetween — R-435 (D7, `09` §3 decision 191). The clean-up windows the hub opened for this
@@ -383,13 +408,17 @@ type WindowCredit struct {
// explains max_remove. The CALLER spends each window once (OffsiteChecker.usedWindows), so the slack
// before `from` cannot let one window explain several falls.
//
// D84: a window that carried ended apps explains, ON TOP of that cap, the ids the hub agreed to (EndedIDs). The cap's
// own rule is unchanged: CapExplains is computed exactly as Explains was, on the fall that is left after the ended
// ids. Pinned by TestD84_WindowCredits_TheEndedAllowanceIsBesideTheCap.
//
// unknown = true only when the store cannot answer (a query error, or a window with no usable cap). The
// caller then falls back to the half-rule — never to „explained". Pinned by r435_windows_between_test.go
// and r435_pinned_drop_test.go.
func (s *Store) WindowCreditsBetween(customerID string, from, to time.Time) (credits []WindowCredit, unknown bool) {
const f = "2006-01-02 15:04:05"
rows, err := s.db.Query(`
SELECT id, count_before, count_after, max_remove, closed_at IS NULL FROM offsite_windows
SELECT id, count_before, count_after, max_remove, closed_at IS NULL, ended_agreed, ended_removed FROM offsite_windows
WHERE customer_id = ?
AND ((closed_at IS NULL AND opened_at <= ? AND closes_by > ?) OR (closed_at > ? AND closed_at <= ?))`,
customerID, to.UTC().Format(f), from.UTC().Format(f), from.UTC().Format(f), to.UTC().Format(f))
@@ -401,7 +430,8 @@ func (s *Store) WindowCreditsBetween(customerID string, from, to time.Time) (cre
var id int64
var before, after, maxRemove sql.NullInt64
var open bool
if err := rows.Scan(&id, &before, &after, &maxRemove, &open); err != nil {
var endAgreed, endRemoved sql.NullString
if err := rows.Scan(&id, &before, &after, &maxRemove, &open, &endAgreed, &endRemoved); err != nil {
return nil, true
}
if !maxRemove.Valid || maxRemove.Int64 <= 0 {
@@ -409,13 +439,20 @@ func (s *Store) WindowCreditsBetween(customerID string, from, to time.Time) (cre
continue
}
explains := int(maxRemove.Int64)
// With no box result the window explains every id the hub agreed to; with one, only those the box named.
ended := jsonStrings(endAgreed)
if !open && before.Valid && after.Valid && after.Int64 >= 0 {
if d := int(before.Int64 - after.Int64); d < explains {
ended = jsonStrings(endRemoved)
d := int(before.Int64-after.Int64) - len(ended) // the fall that is left for the cap to explain
if d < 0 {
d = 0
}
if d < explains {
explains = d
}
}
if explains > 0 {
credits = append(credits, WindowCredit{ID: id, Explains: explains})
if explains > 0 || len(ended) > 0 {
credits = append(credits, WindowCredit{ID: id, Explains: explains + len(ended), CapExplains: explains, EndedIDs: ended})
}
}
if rows.Err() != nil {
+1
View File
@@ -27,6 +27,7 @@ var orphanTables = []struct{ table, where string }{
{"offsite_keys", `customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)`},
{"offsite_windows", `customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)`},
{"offsite_abandon_requests", `customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)`},
{"offsite_app_presence", `customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)`},
}
// PurgeOrphanRows removes, in ONE transaction, the rows of boxes and households that no longer exist from the six
+4
View File
@@ -875,6 +875,10 @@ func (s *Store) migrate() error {
}
// R-833 (v0.129.0): the cap each window was opened with — an operator grant may raise it once.
s.db.Exec("ALTER TABLE offsite_windows ADD COLUMN max_remove INTEGER")
// D72/D84: the hub's own record of which apps a box runs, and what a window agreed to for ended apps.
if err := s.migrateOffsiteEnded(); err != nil {
return err
}
// OS updates, guest fast lane (hub v0.130.0, `11` §8 step 2).
if err := s.migrateOSUpdates(); err != nil {
return fmt.Errorf("os_updates: %w", err)
+26
View File
@@ -236,3 +236,29 @@ func TestR1299_CancelRemovesAWaitingGrant(t *testing.T) {
}
_ = evs
}
// D84 (`07` §8.2f part 4): the window says what the hub agreed to for removed apps and what it refused — per branch of
// the gate. RED-PROOF: documentation/audits/build-release-4-2026-11-15/A/redproof-hub-page.txt.
func TestD84_TheWindowBlockSaysWhatWasAgreedAndRefused(t *testing.T) {
_, st, h, _ := windowRig(t, windowReport)
id, err := st.OpenOffsiteWindowRow("acme", time.Now().Add(20*time.Minute), 40, 20)
if err != nil {
t.Fatal(err)
}
if strings.Contains(windowPage(t, h, ""), `id="window-ended"`) {
t.Error("a window with no removed app must not render the removed-apps line")
}
if err := st.SetOffsiteWindowEnded(id, []string{"a1", "a2", "a3"}, map[string][]string{"vikunja": {"a1", "a2", "a3"}},
[]store.EndedRefusal{{App: "immich", Reason: "the box's newest report lists the app as installed"}}); err != nil {
t.Fatal(err)
}
if _, err := st.SetOffsiteWindowEndedRemoved(id, []string{"a1", "a2"}); err != nil {
t.Fatal(err)
}
page := windowPage(t, h, "")
for _, want := range []string{`id="window-ended"`, "3 snapshot(s) agreed outside the cap, 2 removed", "<strong>refused</strong> immich: the box&#39;s newest report lists the app as installed"} {
if !strings.Contains(page, want) {
t.Errorf("the window block lacks %q", want)
}
}
}
@@ -1061,6 +1061,12 @@
<span class="label">Last window</span>
<span class="value">{{with .Last}}#{{.ID}} {{.OpenedAt.UTC.Format "2006-01-02 15:04"}} UTC &mdash; {{if .ClosedAt.IsZero}}open{{else}}{{.BoxResult}}, {{.CountBefore}} &rarr; {{.CountAfter}}{{end}}{{if .MaxRemove}} (cap {{.MaxRemove}}){{end}}{{else}}never{{end}}</span>
</div>
{{with .Last}}{{if or .EndedAgreed .EndedRefused}}
<div class="info-item" id="window-ended">
<span class="label">Removed apps (30 days after &bdquo;delete backup data&rdquo;)</span>
<span class="value">{{len .EndedAgreed}} snapshot(s) agreed outside the cap, {{len .EndedRemoved}} removed{{range .EndedRefused}}<br><strong>refused</strong>{{if .App}} {{.App}}{{end}}: {{.Reason}}{{end}}</span>
</div>
{{end}}{{end}}
</div>
{{if .Pending}}
<form method="POST" action="/offsite/window-grant/{{$.CustomerID}}" class="inline-confirm">