hub v0.127.0: off-site key registrar (box never gets the storage password), password sealed at rest, daily key check, clean-up window (shipped off) — decisions 68-69, R-820/R-821/R-822
gates / gates (push) Successful in 29s
gates / gates (push) Successful in 29s
Part A evidence (migration spike, sftp-written repo through the pinned rclone key) and the hub red-proofs under documentation/audits/offsite-lock-build-2026-10-03/. Manifest bump follows after the image is built and Secret/offsite-secret-key exists. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -67,6 +67,14 @@
|
||||
| `compareVersions` | hub/internal/web/server.go (~L571) | `(a, b string) int` | X.Y.Z comparisons in web (floor checks, update-available) | Returns 0 on parse error — unparseable compares as "equal" (see §3). |
|
||||
| `reportBackupCard` + `backupCardView` + `fmtBytesAuto` (R-331, v0.109.0) | hub/internal/web/backup_card.go | `(reportJSON string) backupCardView` / `(int64) string` | THE customer-page Backup card — everything it shows about a customer's backups | **Reads the report's `offsite` object, NEVER `backup`.** The `backup` object's `snapshot_count`/`repo_size_mb`/`integrity_ok` have had no producer since slice 8C and rendering them told every operator every customer had `Snapshots 0` (measured on demo-hp over a repo holding 67). **Always consult `StatsKnown` before believing a zero** — absent/false means "never measured", NOT "empty", and those are opposite news (R-225 measured the same confusion one layer down). Resolved in Go, not the template, because a `{{if}}` chain over `Report`'s `map[string]interface{}` float64s cannot keep the absent/zero distinction the card is entirely about. `fmtBytesAuto` scales MB/GB/TB — do NOT swap in `fmtBytesGB`, which renders demo-hp's real 140 829 678 B as `0.1 GB`. |
|
||||
|
||||
### Off-site key registrar (v0.127.0, decisions 68–69, hub/internal/offsitekeys + store)
|
||||
|
||||
| Symbol | File | Short signature | Use for | Gotchas |
|
||||
|---|---|---|---|---|
|
||||
| `offsitekeys.Registrar` (`Install` / `Confirm` / `Audit` / `OpenWindow` / `CloseWindow` / `MoveAside`) | hub/internal/offsitekeys/offsitekeys.go | `(ctx, Target, password, …)` | EVERY write to a sub-account's `.ssh/authorized_keys` and every repo move-aside | **The only writer of that file.** Uses the provider's port-23 restricted shell (`dd of=` takes stdin, `mv` overwrites, `test` does NOT exist — measured); an unpinned line is a deletion route and is dropped on every install; the window line goes FIRST (first match wins). Never `rm`. |
|
||||
| `offsitekeys.Service` (`RegisterKey`, `ConfirmKey`, `AuditAll`, `OpenWindowFor`, `CloseWindowFor`, `SweepExpiredWindows`) | hub/internal/offsitekeys/service.go | — | Binding the registrar to the store, descriptor and operator events | The box-facing API (`/api/v1/offsite/register-key…`) answers with NO credential — pinned by `TestOffsiteKeyEndpoints_AuthAndNoPasswordInAnyResponse`. |
|
||||
| `(*Store).SaveOneTimeSecret` / `OffsitePassword` / `SealLegacyOffsiteSecrets` | hub/internal/store/offsite_seal.go | — | Storing / reading the sub-account password | **Sealed AES-256-GCM; no key → refused (fail-closed).** Under `go test` every store gets a fixed key (`testing.Testing()`); production needs `OFFSITE_SECRET_KEY`. Never serve the value to a box. |
|
||||
|
||||
### Host views & lifecycle / offsite endpoints (v0.47.0, hub/internal/web + store)
|
||||
|
||||
| Symbol | File | Short signature | Use for | Gotchas |
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
## 1. TODAY's transport (sftp:, unpinned key)
|
||||
$ sftp.sh init
|
||||
created restic repository 8becf8f1de at sftp:u629488-sub4@u629488-sub4.your-storagebox.de:spike-migrate
|
||||
|
||||
Please note that knowledge of your password is required to access
|
||||
the repository. Losing your password means that your data is
|
||||
irrecoverably lost.
|
||||
[rc=0]
|
||||
|
||||
$ sftp.sh backup /d/tree --host migbox --tag app1
|
||||
no parent snapshot found, will read all files
|
||||
|
||||
Files: 3 new, 0 changed, 0 unmodified
|
||||
Dirs: 3 new, 0 changed, 0 unmodified
|
||||
Added to the repository: 392.552 KiB (392.119 KiB stored)
|
||||
|
||||
processed 3 files, 390.639 KiB in 0:02
|
||||
snapshot 9c767903 saved
|
||||
[rc=0]
|
||||
|
||||
$ sftp.sh backup /d/tree --host migbox --tag app1
|
||||
using parent snapshot 9c767903
|
||||
|
||||
Files: 0 new, 1 changed, 2 unmodified
|
||||
Dirs: 0 new, 3 changed, 0 unmodified
|
||||
Added to the repository: 1.929 KiB (1.328 KiB stored)
|
||||
|
||||
processed 3 files, 390.641 KiB in 0:01
|
||||
snapshot 761dc658 saved
|
||||
[rc=0]
|
||||
|
||||
$ sftp.sh snapshots
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
9c767903 2026-10-03 14:04:12 migbox app1 /d/tree
|
||||
761dc658 2026-10-03 14:04:18 migbox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
2 snapshots
|
||||
[rc=0]
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
## 2. key line now PINNED (same key). 3. rclone: through it
|
||||
control: today's sftp: transport through the pinned key:
|
||||
$ sftp.sh snapshots
|
||||
Fatal: unable to open repository at sftp:u629488-sub4@u629488-sub4.your-storagebox.de:spike-migrate: unable to start the sftp session, error: unexpected EOF
|
||||
[rc=1]
|
||||
|
||||
$ rc.sh snapshots
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
9c767903 2026-10-03 14:04:12 migbox app1 /d/tree
|
||||
761dc658 2026-10-03 14:04:18 migbox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
2 snapshots
|
||||
[rc=0]
|
||||
|
||||
$ rc.sh backup /d/tree --host migbox --tag app1
|
||||
using parent snapshot 761dc658
|
||||
|
||||
Files: 0 new, 0 changed, 3 unmodified
|
||||
Dirs: 0 new, 0 changed, 3 unmodified
|
||||
Added to the repository: 0 B (0 B stored)
|
||||
|
||||
processed 3 files, 390.641 KiB in 0:00
|
||||
snapshot 56667008 saved
|
||||
[rc=0]
|
||||
|
||||
$ rc.sh restore 9c767903 --target /d/restored --include /d/tree/sub/note.txt
|
||||
restoring <Snapshot 9c767903 of [/d/tree] at 2026-10-03 14:04:12.212018962 +0000 UTC by root@migbox> to /d/restored
|
||||
[rc=0]
|
||||
|
||||
restored sftp-era file == its content at that snapshot: IDENTICAL
|
||||
$ rc.sh check
|
||||
using temporary cache in /tmp/restic-check-cache-456083153
|
||||
create exclusive lock for repository
|
||||
load indexes
|
||||
check all packs
|
||||
check snapshots, trees and blobs
|
||||
[0:00] 100.00% 3 / 3 snapshots
|
||||
|
||||
no errors were found
|
||||
[rc=0]
|
||||
|
||||
$ rc.sh check --read-data
|
||||
using temporary cache in /tmp/restic-check-cache-2313441632
|
||||
create exclusive lock for repository
|
||||
load indexes
|
||||
check all packs
|
||||
check snapshots, trees and blobs
|
||||
[0:00] 100.00% 3 / 3 snapshots
|
||||
|
||||
read all data
|
||||
[0:00] 100.00% 4 / 4 packs
|
||||
|
||||
no errors were found
|
||||
[rc=0]
|
||||
|
||||
## 4.
|
||||
$ rc.sh forget 9c767903
|
||||
unable to remove <snapshot/9c767903c1> from the repository
|
||||
[0:48] 0.00% 0 / 1 files deleted
|
||||
|
||||
blob not removed, server response: 403 Forbidden (403)
|
||||
[rc=1]
|
||||
|
||||
$ rc.sh snapshots
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
9c767903 2026-10-03 14:04:12 migbox app1 /d/tree
|
||||
761dc658 2026-10-03 14:04:18 migbox app1 /d/tree
|
||||
56667008 2026-10-03 14:04:51 migbox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
3 snapshots
|
||||
[rc=0]
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
## E1: SAME key twice — append-only line FIRST, deleting line SECOND
|
||||
$ rc.sh forget 56667008
|
||||
unable to remove <snapshot/5666700865> from the repository
|
||||
[0:48] 0.00% 0 / 1 files deleted
|
||||
|
||||
blob not removed, server response: 403 Forbidden (403)
|
||||
[rc=1]
|
||||
|
||||
## E2: SAME key twice — deleting line FIRST
|
||||
$ rc.sh forget 56667008 --dry-run
|
||||
Would have removed the following snapshots:
|
||||
{56667008}
|
||||
|
||||
[rc=0]
|
||||
|
||||
## E3: a SECOND key (window key) on its own deleting line, the box key stays pinned
|
||||
$ rcw.sh forget 56667008
|
||||
[0:00] 100.00% 1 / 1 files deleted
|
||||
|
||||
[rc=0]
|
||||
|
||||
$ rc.sh snapshots
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
9c767903 2026-10-03 14:04:12 migbox app1 /d/tree
|
||||
761dc658 2026-10-03 14:04:18 migbox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
2 snapshots
|
||||
[rc=0]
|
||||
|
||||
## E3b: pinned key still refused while the window key exists
|
||||
$ rc.sh forget 761dc658
|
||||
[0:48] 0.00% 0 / 1 files deleted
|
||||
|
||||
unable to remove <snapshot/761dc6583f> from the repository
|
||||
blob not removed, server response: 403 Forbidden (403)
|
||||
[rc=1]
|
||||
|
||||
## E2 (real): SAME key, deleting line FIRST — a real forget
|
||||
$ rc.sh forget 761dc658
|
||||
[0:00] 100.00% 1 / 1 files deleted
|
||||
|
||||
[rc=0]
|
||||
|
||||
$ rc.sh snapshots
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
9c767903 2026-10-03 14:04:12 migbox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
1 snapshots
|
||||
[rc=0]
|
||||
|
||||
## E2 closed: deleting line removed again -> refused
|
||||
$ rc.sh forget 9c767903
|
||||
[0:48] 0.00% 0 / 1 files deleted
|
||||
|
||||
unable to remove <snapshot/9c767903c1> from the repository
|
||||
blob not removed, server response: 403 Forbidden (403)
|
||||
[rc=1]
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
## Part A teardown 2026-10-03T14:09:25Z
|
||||
authorized_keys sha256 now 795e715315973740 / orig 795e715315973740
|
||||
home: . .. .config .ssh felhom-repo spike-migrate
|
||||
spike-migrate KEPT on purpose (1 snapshot) for Part E's planted history; .config/rclone is the provider rclone's
|
||||
@@ -0,0 +1,12 @@
|
||||
## pinned path ABSOLUTE (/home/spike-migrate, the descriptor's form)
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
9c767903 2026-10-03 14:04:12 migbox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
1 snapshots
|
||||
## probe: ssh with the pinned key, stdin closed
|
||||
2026/10/03 14:11:14 NOTICE: Config file "/home/.config/rclone/rclone.conf" not found - using defaults
|
||||
rc=0
|
||||
## probe with an UNPINNED key (control)
|
||||
Command not found. Use 'help' to get a list of available commands.
|
||||
rc=8
|
||||
@@ -0,0 +1,9 @@
|
||||
## cat config on a repo that does not exist (pinned key)
|
||||
Fatal: unable to open config file: <config/> does not exist
|
||||
Is there a repository at the following location?
|
||||
rclone:spike-migrate
|
||||
rc=1
|
||||
## wrong key (window key not in the file) — transport failure shape
|
||||
rclone: u629488-sub4@u629488-sub4.your-storagebox.de: Permission denied (publickey,password).
|
||||
Fatal: unable to open repository at rclone:spike-migrate: error talking HTTP to rclone: Get "http://localhost/file-5577006791947779410": unexpected EOF
|
||||
rc=1
|
||||
@@ -0,0 +1,12 @@
|
||||
# Part A exit test — written before any command (2026-10-03 evening)
|
||||
|
||||
Venue: `u629488-sub4` (tester-1), repo dir `spike-migrate` only. restic 0.14.0 from controller image 0.288.0.
|
||||
|
||||
1. A repo is created and backed up TODAY'S way: `sftp:` transport, an UNPINNED key, port 23 — 2 snapshots.
|
||||
2. The key line is then replaced by the PINNED line (`command="rclone serve restic --stdio --append-only spike-migrate",restrict`).
|
||||
3. Through the pinned key over `rclone:` (`-o rclone.program="ssh -p 23 … -i <key> … rclone"`), MUST succeed:
|
||||
`snapshots` (both sftp-era snapshots listed), `backup` (count 2 → 3, parent = the sftp-era snapshot),
|
||||
`restore` of one file from an sftp-era snapshot (bytes identical), `check` (exclusive lock taken and released),
|
||||
`check --read-data` (the integrity job's full depth).
|
||||
4. Through the pinned key MUST be refused: `forget <sftp-era id>` (403). Count stays 3.
|
||||
5. Fail → stop and report; no build.
|
||||
@@ -0,0 +1,22 @@
|
||||
## RP1: SaveOneTimeSecret without sealing (the pre-v0.127.0 behaviour)
|
||||
=== RUN TestOffsiteSecret_RawRowHoldsNoPassword
|
||||
offsite_secret_seal_test.go:39: raw row is not sealed: "Sup3rSecretPw%"
|
||||
--- FAIL: TestOffsiteSecret_RawRowHoldsNoPassword (0.02s)
|
||||
|
||||
## RP2: consume handler serving the password again (the pre-v0.127.0 handler)
|
||||
=== RUN TestConsumePassword_RetiredReturnsNoPassword
|
||||
offsite_test.go:43: consume → 200, want 410 (retired)
|
||||
--- FAIL: TestConsumePassword_RetiredReturnsNoPassword (0.02s)
|
||||
|
||||
## RP3: audit that ignores the pin (every line counted as pinned)
|
||||
=== RUN TestInstall_MigratesUnpinnedKeyAndAuditGoesClean
|
||||
offsitekeys_test.go:98: before: {Lines:2 Pinned:2 Findings:[]} <nil> — want 2 unpinned findings (the decoy must be seen)
|
||||
--- FAIL: TestInstall_MigratesUnpinnedKeyAndAuditGoesClean (0.00s)
|
||||
=== RUN TestWindow_PrependAuditClose
|
||||
offsitekeys_test.go:171: a window line with no open window must alarm: {Lines:2 Pinned:2 Findings:[]}
|
||||
--- FAIL: TestWindow_PrependAuditClose (0.00s)
|
||||
|
||||
## restored — all green:
|
||||
ok gitea.dooplex.hu/admin/felhom-hub/internal/store 2.888s
|
||||
ok gitea.dooplex.hu/admin/felhom-hub/internal/api 4.047s
|
||||
ok gitea.dooplex.hu/admin/felhom-hub/internal/offsitekeys 0.009s
|
||||
@@ -0,0 +1,21 @@
|
||||
## RPC1: retention ignores the pin (the pre-v0.289.0 'forget --prune after every run')
|
||||
=== RUN TestRetention_PinnedWithoutWindowDeletesNothing
|
||||
offbox_window_test.go:106: a forget ran without a window: [[forget --group-by host,tags --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune] [forget --group-by host,tags --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune]]
|
||||
--- FAIL: TestRetention_PinnedWithoutWindowDeletesNothing (0.00s)
|
||||
=== RUN TestRunOffboxBackup_PinnedNeverForgetsWithoutWindow
|
||||
offbox_window_test.go:127: the pinned run reached forget 1 time(s)
|
||||
--- FAIL: TestRunOffboxBackup_PinnedNeverForgetsWithoutWindow (0.00s)
|
||||
|
||||
## RPC2: guard without the future-date check
|
||||
=== RUN TestOffsiteGuard_LabThirteenFutureFakes_Refused
|
||||
offbox_window_test.go:154: window close = [{ID:7 CountBefore:16 CountAfter:16 Removed:0 Outcome:guard-refused Reason:the policy would remove snapshot r1 from 2026-10-03T12:45:38Z — younger than 8 days, which honest retention never does}]
|
||||
--- FAIL: TestOffsiteGuard_LabThirteenFutureFakes_Refused (0.00s)
|
||||
|
||||
## RPC3: bridge trusts the registrar without proving the pin
|
||||
=== RUN TestBridge_RegisteredButNotPinnedRefuses
|
||||
offsiteapply_test.go:212: a key that does not reach the pinned server must refuse
|
||||
--- FAIL: TestBridge_RegisteredButNotPinnedRefuses (0.00s)
|
||||
|
||||
## restored:
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 433.945s
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/offsiteapply (cached)
|
||||
@@ -117,6 +117,29 @@ shred -u /path/to/keyfile
|
||||
|
||||
---
|
||||
|
||||
## Off-site password sealing key — `Secret/offsite-secret-key` (hub v0.127.0, decision 69, R-821)
|
||||
|
||||
**What uses it:** `hub` env `OFFSITE_SECRET_KEY` (64 hex characters = 32 bytes). It seals every Storage Box
|
||||
sub-account password in `one_time_secrets.value`; the hub's key registrar opens them to write box keys.
|
||||
**Required** — the pod does not start without it.
|
||||
|
||||
**Create (once, before the first v0.127.0 sync) — the value never touches a file or the terminal:**
|
||||
|
||||
```bash
|
||||
sudo kubectl -n felhom-system create secret generic offsite-secret-key \
|
||||
--from-literal=OFFSITE_SECRET_KEY="$(openssl rand -hex 32)"
|
||||
```
|
||||
|
||||
**If it is lost:** the sealed passwords cannot be opened. Nothing on the boxes breaks (their keys are installed);
|
||||
the registrar and the daily check fail with `offsite_key_audit_failed`. Recover per customer with the hub's
|
||||
**Re-issue offsite credentials** button (the provider resets the password; the hub seals the new one). Keep a copy
|
||||
in the operator's password manager if a Re-issue round is not acceptable.
|
||||
|
||||
**Rotation:** not built. A new key cannot open the old rows; rotate by setting the new key and pressing Re-issue
|
||||
for every off-site customer.
|
||||
|
||||
---
|
||||
|
||||
## Other committed secrets (tracked, NOT yet de-gitted — backlog)
|
||||
|
||||
`manifests/felhom.secret.yaml` still commits other plaintext secrets (`healthchecks-config` `SECRET_KEY`
|
||||
|
||||
@@ -1,3 +1,30 @@
|
||||
## v0.127.0 — off-site keys a box cannot use to delete: the hub is the key registrar, the storage password is sealed and never served, a daily key check, the clean-up window (decisions 68–69, R-820, R-821, R-822) (2026-10-03)
|
||||
|
||||
- **The box never receives the Storage Box sub-account password again (R-820).** `POST /api/v1/offsite/consume-password/`
|
||||
answers **410** with no body that could carry it. A box sends its PUBLIC key to `POST /api/v1/offsite/register-key/<id>`;
|
||||
the hub (new `internal/offsitekeys`) writes it into the sub-account's `.ssh/authorized_keys` pinned to
|
||||
`command="rclone serve restic --stdio --append-only <repo>",restrict` over the provider's port-23 restricted shell
|
||||
(`dd`/`mv`/`cat`, measured — no new dependency), drops every UNPINNED line in the same write, reads the file back, and
|
||||
records the key. `confirm-key` keeps only the confirmed key (rotation: write new → box confirms → remove old).
|
||||
`move-aside` renames an orphaned repository (never deletes) for the box. The off-site self-heal's message no longer
|
||||
says the box "re-consumes" a password.
|
||||
- **The password is sealed at rest (R-821).** `one_time_secrets.value` is AES-256-GCM (`enc:v1:`) under
|
||||
`OFFSITE_SECRET_KEY` from the out-of-band `Secret/offsite-secret-key`; legacy plaintext rows are sealed at start-up.
|
||||
Without the key the hub refuses to save or use any sub-account password (fail-closed) and the registrar is off.
|
||||
- **The daily key check (07:10 Budapest; on demand `POST /offsite/key-audit`):** every provisioned sub-account's
|
||||
`authorized_keys` is read; any line that can delete outside an open window raises `offsite_key_unlocked` (error,
|
||||
operator-only, the line named by fingerprint only); an unreadable file raises `offsite_key_audit_failed`.
|
||||
- **The clean-up window (decision 68), shipped OFF.** `window-open` grants a box at most one 20-minute window a week
|
||||
(operator switch `POST /offsite/windows-enabled`, default off) or on an operator one-shot (`POST /offsite/window-grant/<id>`),
|
||||
by PREPENDING a deleting line for the box's confirmed key (first match wins — measured); `window-close` removes it and
|
||||
alarms `offsite_window_drop` when the count fell by more than `MaxRemove` (40 %, ≥ 5), `offsite_prune_guard_refused`
|
||||
when the box's fake-snapshot guard refused (R-822), `offsite_window_failed` on an error; a window not closed in 20 min
|
||||
is closed by the hub. New operator-only events registered in the same commit; `offbox_abandon_deferred` allow-listed.
|
||||
- Tests: `TestConsumePassword_RetiredReturnsNoPassword`, `TestOffsiteKeyEndpoints_AuthAndNoPasswordInAnyResponse`,
|
||||
`TestOffsiteSecret_*`, `TestSealLegacyOffsiteSecrets_*`, `internal/offsitekeys` (migration, rotation, window, move-aside,
|
||||
audit decoy) — red-proofs in `documentation/audits/offsite-lock-build-2026-10-03/partB/`.
|
||||
- **Deploy order:** create `Secret/offsite-secret-key` BEFORE syncing (the manifest makes it required).
|
||||
|
||||
## v0.126.0 — a fresh connect link from the old one (R-719); no "recovered" for a new box and no first-hour tier-skip mail (R-723); the bind page names who hands over the phrase (R-725) (2026-09-30)
|
||||
|
||||
- **R-719:** a box that registers is UNCLAIMED — the registration carries uuid, MACs, host keys and hardware, nothing
|
||||
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsiteheal"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsitekeys"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/pbsdrheal"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/poke"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
@@ -179,6 +180,26 @@ func main() {
|
||||
logger.Fatalf("[FATAL] Failed to initialize store: %v", err)
|
||||
}
|
||||
defer dataStore.Close()
|
||||
|
||||
// R-821 / decision 69 (v0.127.0): the off-site sub-account password is sealed at rest with a key
|
||||
// that is NOT in the database (Secret/offsite-secret-key). Without it the hub cannot store, read or
|
||||
// use any sub-account password — provisioning and the key registrar refuse (fail-closed), and the
|
||||
// legacy plaintext rows stay as they are until the key arrives.
|
||||
offsiteKeyReady := false
|
||||
if v := os.Getenv("OFFSITE_SECRET_KEY"); v == "" {
|
||||
logger.Printf("[ERROR] OFFSITE_SECRET_KEY unset — off-site passwords cannot be sealed; provisioning and the key registrar are DISABLED")
|
||||
} else if key, kerr := store.ParseOffsiteSecretKey(v); kerr != nil {
|
||||
logger.Printf("[ERROR] OFFSITE_SECRET_KEY invalid (%v) — provisioning and the key registrar are DISABLED", kerr)
|
||||
} else if kerr := dataStore.SetOffsiteSecretKey(key); kerr != nil {
|
||||
logger.Printf("[ERROR] OFFSITE_SECRET_KEY rejected (%v) — provisioning and the key registrar are DISABLED", kerr)
|
||||
} else {
|
||||
offsiteKeyReady = true
|
||||
if n, serr := dataStore.SealLegacyOffsiteSecrets(); serr != nil {
|
||||
logger.Printf("[ERROR] sealing legacy off-site secrets failed after %d row(s): %v", n, serr)
|
||||
} else {
|
||||
logger.Printf("[INFO] off-site secrets sealed at rest (%d legacy plaintext row(s) sealed now)", n)
|
||||
}
|
||||
}
|
||||
logger.Printf("[INFO] Database opened at %s", dbPath)
|
||||
|
||||
// Phase 2 managed updates: seed the global controller-version floor fallback (config/env). A
|
||||
@@ -363,6 +384,60 @@ func main() {
|
||||
// unconfigured or the customer has no offsite tier.
|
||||
apiHandler.SetOffsiteReissuer(webServer.ReissueOffsiteForCustomer)
|
||||
|
||||
// Decision 69 (v0.127.0): the off-site KEY REGISTRAR. The box sends its public key; the hub writes it
|
||||
// into the sub-account's authorized_keys pinned append-only; the daily check reads every file.
|
||||
if offsiteKeyReady {
|
||||
keySvc := &offsitekeys.Service{
|
||||
Store: dataStore, Reg: &offsitekeys.Registrar{Dialer: offsitekeys.SSHDialer{}}, Logger: logger,
|
||||
Emit: dispatcher.ProcessEvent,
|
||||
}
|
||||
apiHandler.SetOffsiteKeyService(keySvc)
|
||||
runKeyAudit := func(ctx context.Context) any {
|
||||
start := time.Now()
|
||||
out := keySvc.AuditAll(ctx, dataStore.OffsiteWindowOpen)
|
||||
logger.Printf("[INFO] off-site key check: %d sub-account(s) read in %s", len(out), time.Since(start).Round(time.Millisecond))
|
||||
type row struct {
|
||||
Customer string `json:"customer"`
|
||||
Lines int `json:"lines"`
|
||||
Pinned int `json:"pinned"`
|
||||
Findings []offsitekeys.Finding `json:"findings"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
var rows []row
|
||||
for _, o := range out {
|
||||
rr := row{Customer: o.CustomerID, Lines: o.Result.Lines, Pinned: o.Result.Pinned, Findings: o.Result.Findings}
|
||||
if o.Err != nil {
|
||||
rr.Error = o.Err.Error()
|
||||
}
|
||||
rows = append(rows, rr)
|
||||
}
|
||||
return rows
|
||||
}
|
||||
webServer.SetOffsiteKeyAudit(runKeyAudit)
|
||||
webServer.SetOffsiteWindowAdmin(dataStore.GrantOffsiteWindowOnce, dataStore.SetOffsiteWindowsEnabled)
|
||||
// Decision 68: a window the box never closed is closed by the hub at its 20-minute bound.
|
||||
go func() {
|
||||
tk := time.NewTicker(60 * time.Second)
|
||||
defer tk.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-tk.C:
|
||||
sctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
|
||||
keySvc.SweepExpiredWindows(sctx)
|
||||
cancel()
|
||||
}
|
||||
}
|
||||
}()
|
||||
go scheduleDaily(ctx, "offsite-key-audit", "07:10", func() {
|
||||
actx, cancel := context.WithTimeout(ctx, 10*time.Minute)
|
||||
defer cancel()
|
||||
runKeyAudit(actx)
|
||||
}, logger)
|
||||
logger.Printf("[INFO] Off-site key registrar enabled; daily key check at 07:10 Budapest")
|
||||
}
|
||||
|
||||
// Direction-2 immediate-sync (v0.58.0): one in-memory operator-intent notifier, shared by the
|
||||
// web handlers (which Bump it after every intent write) and the API handler (which long-polls it
|
||||
// at GET /api/v1/wait). In-memory BY DESIGN — a restart resets generations to zero; the box
|
||||
|
||||
@@ -49,6 +49,7 @@ type Poker interface {
|
||||
// Handler handles API endpoints for report ingest and customer queries.
|
||||
type Handler struct {
|
||||
store *store.Store
|
||||
offsiteKeys OffsiteKeyService // decision 69 key registrar (nil → 503)
|
||||
apiKey string
|
||||
resendAPIKey string
|
||||
fromEmail string
|
||||
@@ -355,6 +356,16 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
case r.Method == http.MethodPost && strings.HasPrefix(path, "/offsite/consume-password/"):
|
||||
customerID := strings.TrimPrefix(path, "/offsite/consume-password/")
|
||||
h.handleOffsiteConsumePassword(w, r, customerID)
|
||||
case r.Method == http.MethodPost && strings.HasPrefix(path, "/offsite/register-key/"):
|
||||
h.handleOffsiteRegisterKey(w, r, strings.TrimPrefix(path, "/offsite/register-key/"))
|
||||
case r.Method == http.MethodPost && strings.HasPrefix(path, "/offsite/confirm-key/"):
|
||||
h.handleOffsiteConfirmKey(w, r, strings.TrimPrefix(path, "/offsite/confirm-key/"))
|
||||
case r.Method == http.MethodPost && strings.HasPrefix(path, "/offsite/move-aside/"):
|
||||
h.handleOffsiteMoveAside(w, r, strings.TrimPrefix(path, "/offsite/move-aside/"))
|
||||
case r.Method == http.MethodPost && strings.HasPrefix(path, "/offsite/window-open/"):
|
||||
h.handleOffsiteWindowOpen(w, r, strings.TrimPrefix(path, "/offsite/window-open/"))
|
||||
case r.Method == http.MethodPost && strings.HasPrefix(path, "/offsite/window-close/"):
|
||||
h.handleOffsiteWindowClose(w, r, strings.TrimPrefix(path, "/offsite/window-close/"))
|
||||
case r.Method == http.MethodGet && strings.HasPrefix(path, "/artifacts/"):
|
||||
customerID := strings.TrimPrefix(path, "/artifacts/")
|
||||
h.handleArtifactManifest(w, r, customerID)
|
||||
@@ -2049,8 +2060,11 @@ var allowedEventTypes = map[string]bool{
|
||||
"offsite_proof_empty": true,
|
||||
// R-431 — the hub raises this itself; allowlisted so a hub-origin event is never 400'd.
|
||||
"offsite_snapshots_dropped": true,
|
||||
"crossdrive_completed": true,
|
||||
"crossdrive_failed": true,
|
||||
// controller v0.289.0 (decision 69): the customer-chosen deletion of set-aside history is deferred
|
||||
// to the operator — the box's append-only key cannot delete. Operator-only (notify.operatorOnlyEvents).
|
||||
"offbox_abandon_deferred": true,
|
||||
"crossdrive_completed": true,
|
||||
"crossdrive_failed": true,
|
||||
// controller v0.134.1 — enlarged offsite push refused by the quota gate (warning; the controller's
|
||||
// dynamic Hungarian message is customer-grade — deliberately NO customerMessages entry, which would
|
||||
// discard the numbers (templates.go:129 priority)).
|
||||
|
||||
+154
-17
@@ -1,31 +1,168 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsitekeys"
|
||||
)
|
||||
|
||||
// handleOffsiteConsumePassword serves the one-time transient offsite password to the controller EXACTLY
|
||||
// ONCE (SLICE 1; SLICE 2 controller consumes it, installs its key, then the hub resets the box password).
|
||||
// Auth = the customer's API key (same credential as config-pull); the token's customer must match the
|
||||
// path. The value is returned once then marked consumed — a second call 404s. NEVER logged.
|
||||
// OffsiteKeyService is the registrar seam (decision 69). nil → the key endpoints answer 503.
|
||||
type OffsiteKeyService interface {
|
||||
RegisterKey(ctx context.Context, customerID, pub string) (offsitekeys.InstallResult, error)
|
||||
ConfirmKey(ctx context.Context, customerID, fp string) (int, error)
|
||||
MoveAside(ctx context.Context, customerID string) (string, error)
|
||||
OpenWindowFor(ctx context.Context, customerID string, countBefore int) (offsitekeys.WindowGrant, error)
|
||||
CloseWindowFor(ctx context.Context, customerID string, r offsitekeys.WindowResult) error
|
||||
}
|
||||
|
||||
// SetOffsiteKeyService wires the key registrar.
|
||||
func (h *Handler) SetOffsiteKeyService(s OffsiteKeyService) { h.offsiteKeys = s }
|
||||
|
||||
// handleOffsiteConsumePassword is RETIRED (hub v0.127.0, decision 69, R-820). It used to hand the box the
|
||||
// Storage Box sub-account password; that password can rewrite `.ssh/authorized_keys` and so remove the
|
||||
// append-only pin from any key (measured 2026-10-03). A box now sends its PUBLIC key to
|
||||
// /offsite/register-key and the hub installs it. This answers 410 with no body that could carry a secret.
|
||||
// Pinned by TestConsumePassword_RetiredReturnsNoPassword.
|
||||
func (h *Handler) handleOffsiteConsumePassword(w http.ResponseWriter, r *http.Request, customerID string) {
|
||||
authCustomerID, isGlobal, ok := h.checkAuthCustomer(r)
|
||||
if !ok || (!isGlobal && authCustomerID != customerID) {
|
||||
http.Error(w, "unauthorized", http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
pw, err := h.store.ConsumeOneTimeSecret(customerID)
|
||||
if err == sql.ErrNoRows {
|
||||
http.Error(w, "no unconsumed offsite password", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
h.logger.Printf("[ERROR] offsite consume-password %s: %v", customerID, err) // no secret
|
||||
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"password": pw}) // one-time; never logged
|
||||
h.logger.Printf("[WARN] offsite consume-password called by %s — retired (decision 69); the box must register its public key (controller >= 0.289.0)", customerID)
|
||||
http.Error(w, "gone: the hub no longer serves the storage password; register the box's public key at /api/v1/offsite/register-key/", http.StatusGone)
|
||||
}
|
||||
|
||||
func (h *Handler) offsiteKeyAuth(w http.ResponseWriter, r *http.Request, customerID string) bool {
|
||||
authCustomerID, isGlobal, ok := h.checkAuthCustomer(r)
|
||||
if !ok || (!isGlobal && authCustomerID != customerID) {
|
||||
http.Error(w, "unauthorized", http.StatusUnauthorized)
|
||||
return false
|
||||
}
|
||||
if h.offsiteKeys == nil {
|
||||
http.Error(w, "offsite key registrar not configured", http.StatusServiceUnavailable)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func offsiteKeyErr(w http.ResponseWriter, err error) {
|
||||
if errors.Is(err, offsitekeys.ErrNotProvisioned) {
|
||||
http.Error(w, "no provisioned off-site target", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
http.Error(w, "registrar failed: "+err.Error(), http.StatusBadGateway)
|
||||
}
|
||||
|
||||
// handleOffsiteRegisterKey: POST {"public_key": "ssh-ed25519 AAAA… comment"} → the hub writes it into the
|
||||
// sub-account's authorized_keys pinned append-only and answers {"installed":true,"fingerprint":"SHA256:…"}.
|
||||
// The response carries NO credential.
|
||||
func (h *Handler) handleOffsiteRegisterKey(w http.ResponseWriter, r *http.Request, customerID string) {
|
||||
if !h.offsiteKeyAuth(w, r, customerID) {
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
PublicKey string `json:"public_key"`
|
||||
}
|
||||
body, _ := io.ReadAll(io.LimitReader(r.Body, 16<<10))
|
||||
if err := json.Unmarshal(body, &req); err != nil || req.PublicKey == "" {
|
||||
http.Error(w, "body must be {\"public_key\": \"…\"}", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if _, _, err := offsitekeys.KeyFingerprint(req.PublicKey); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Minute)
|
||||
defer cancel()
|
||||
res, err := h.offsiteKeys.RegisterKey(ctx, customerID, req.PublicKey)
|
||||
if err != nil {
|
||||
offsiteKeyErr(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"installed": true, "fingerprint": res.Fingerprint})
|
||||
}
|
||||
|
||||
// handleOffsiteConfirmKey: POST {"fingerprint": "SHA256:…"} → only that key's pinned line stays.
|
||||
func (h *Handler) handleOffsiteConfirmKey(w http.ResponseWriter, r *http.Request, customerID string) {
|
||||
if !h.offsiteKeyAuth(w, r, customerID) {
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Fingerprint string `json:"fingerprint"`
|
||||
}
|
||||
body, _ := io.ReadAll(io.LimitReader(r.Body, 4<<10))
|
||||
if err := json.Unmarshal(body, &req); err != nil || req.Fingerprint == "" {
|
||||
http.Error(w, "body must be {\"fingerprint\": \"SHA256:…\"}", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Minute)
|
||||
defer cancel()
|
||||
removed, err := h.offsiteKeys.ConfirmKey(ctx, customerID, req.Fingerprint)
|
||||
if err != nil {
|
||||
offsiteKeyErr(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"confirmed": true, "removed": removed})
|
||||
}
|
||||
|
||||
// handleOffsiteMoveAside: POST → the hub renames the repository to <repo>.orphaned-<date>[-n]. Never deletes.
|
||||
func (h *Handler) handleOffsiteMoveAside(w http.ResponseWriter, r *http.Request, customerID string) {
|
||||
if !h.offsiteKeyAuth(w, r, customerID) {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Minute)
|
||||
defer cancel()
|
||||
to, err := h.offsiteKeys.MoveAside(ctx, customerID)
|
||||
if err != nil {
|
||||
offsiteKeyErr(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"moved_to": to})
|
||||
}
|
||||
|
||||
// handleOffsiteWindowOpen: POST {"count_before": N} → the hub decides (weekly / operator one-shot) and,
|
||||
// if granted, prepends a deleting line for the box's confirmed key for 20 minutes (decision 68).
|
||||
func (h *Handler) handleOffsiteWindowOpen(w http.ResponseWriter, r *http.Request, customerID string) {
|
||||
if !h.offsiteKeyAuth(w, r, customerID) {
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
CountBefore int `json:"count_before"`
|
||||
}
|
||||
body, _ := io.ReadAll(io.LimitReader(r.Body, 4<<10))
|
||||
_ = json.Unmarshal(body, &req)
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Minute)
|
||||
defer cancel()
|
||||
g, err := h.offsiteKeys.OpenWindowFor(ctx, customerID, req.CountBefore)
|
||||
if err != nil {
|
||||
offsiteKeyErr(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, g)
|
||||
}
|
||||
|
||||
// handleOffsiteWindowClose: POST the box's result → the hub removes the deleting line and checks the count.
|
||||
func (h *Handler) handleOffsiteWindowClose(w http.ResponseWriter, r *http.Request, customerID string) {
|
||||
if !h.offsiteKeyAuth(w, r, customerID) {
|
||||
return
|
||||
}
|
||||
var req offsitekeys.WindowResult
|
||||
body, _ := io.ReadAll(io.LimitReader(r.Body, 8<<10))
|
||||
if err := json.Unmarshal(body, &req); err != nil || req.WindowID == 0 {
|
||||
http.Error(w, "body must carry window_id", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Minute)
|
||||
defer cancel()
|
||||
if err := h.offsiteKeys.CloseWindowFor(ctx, customerID, req); err != nil {
|
||||
offsiteKeyErr(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"closed": true})
|
||||
}
|
||||
|
||||
@@ -1,7 +1,11 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsitekeys"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
@@ -9,14 +13,16 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// The one-time offsite password is served once to the authenticated customer, then 404s; a wrong/absent or
|
||||
// cross-customer key is rejected.
|
||||
func TestOffsite_ConsumePassword(t *testing.T) {
|
||||
// R-820 / decision 69: the consume endpoint is RETIRED — it answers 410 and its body carries no
|
||||
// password, even with a stored, unconsumed secret and the right key. Before v0.127.0 it returned the
|
||||
// sub-account password, which can rewrite authorized_keys and remove the append-only pin.
|
||||
func TestConsumePassword_RetiredReturnsNoPassword(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ckey", RetrievalPassword: "pp"})
|
||||
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c2", APIKey: "ckey2", RetrievalPassword: "pp2"})
|
||||
st.SaveOneTimeSecret("c1", "the-transient-pw")
|
||||
|
||||
if err := st.SaveOneTimeSecret("c1", "the-transient-pw"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
do := func(token string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest(http.MethodPost, "/api/v1/offsite/consume-password/c1", nil)
|
||||
if token != "" {
|
||||
@@ -26,27 +32,99 @@ func TestOffsite_ConsumePassword(t *testing.T) {
|
||||
h.ServeHTTP(rr, req)
|
||||
return rr
|
||||
}
|
||||
|
||||
if rr := do(""); rr.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("no auth → %d, want 401", rr.Code)
|
||||
}
|
||||
if rr := do("wrongkey"); rr.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("wrong key → %d, want 401", rr.Code)
|
||||
}
|
||||
if rr := do("ckey2"); rr.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("cross-customer key → %d, want 401", rr.Code)
|
||||
}
|
||||
// first (authorized) consume → 200 + the password
|
||||
rr := do("ckey")
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("consume → %d, want 200", rr.Code)
|
||||
if rr.Code != http.StatusGone {
|
||||
t.Fatalf("consume → %d, want 410 (retired)", rr.Code)
|
||||
}
|
||||
var body map[string]string
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &body); err != nil || body["password"] != "the-transient-pw" {
|
||||
t.Fatalf("body = %q (%v)", rr.Body.String(), err)
|
||||
if strings.Contains(rr.Body.String(), "the-transient-pw") {
|
||||
t.Fatalf("the retired endpoint leaked the password: %q", rr.Body.String())
|
||||
}
|
||||
// second consume → 404 (single use)
|
||||
if rr2 := do("ckey"); rr2.Code != http.StatusNotFound {
|
||||
t.Fatalf("second consume → %d, want 404", rr2.Code)
|
||||
// The stored secret is untouched (still usable by the HUB's registrar).
|
||||
if pw, err := st.OffsitePassword("c1"); err != nil || pw != "the-transient-pw" {
|
||||
t.Fatalf("stored credential changed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
type fakeKeySvc struct {
|
||||
gotPub string
|
||||
gotFP string
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeKeySvc) RegisterKey(_ context.Context, _ string, pub string) (offsitekeys.InstallResult, error) {
|
||||
f.gotPub = pub
|
||||
return offsitekeys.InstallResult{Fingerprint: "SHA256:fake"}, f.err
|
||||
}
|
||||
func (f *fakeKeySvc) ConfirmKey(_ context.Context, _ string, fp string) (int, error) {
|
||||
f.gotFP = fp
|
||||
return 1, f.err
|
||||
}
|
||||
func (f *fakeKeySvc) OpenWindowFor(context.Context, string, int) (offsitekeys.WindowGrant, error) {
|
||||
return offsitekeys.WindowGrant{Granted: false, Reason: "not due"}, f.err
|
||||
}
|
||||
func (f *fakeKeySvc) CloseWindowFor(context.Context, string, offsitekeys.WindowResult) error { return f.err }
|
||||
func (f *fakeKeySvc) MoveAside(context.Context, string) (string, error) {
|
||||
return "/home/felhom-repo.orphaned-20261003", f.err
|
||||
}
|
||||
|
||||
const testPub = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK436vIXGqfs6wz4Jv/GIIo3rQuW3oNnP7nMatI92gkA box"
|
||||
|
||||
// Every box-facing off-site key response: auth enforced, and NO response body carries the stored
|
||||
// password (the decision-69 invariant, asserted on the consequence — the bytes the box receives).
|
||||
func TestOffsiteKeyEndpoints_AuthAndNoPasswordInAnyResponse(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ckey", RetrievalPassword: "pp"})
|
||||
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c2", APIKey: "ckey2", RetrievalPassword: "pp2"})
|
||||
if err := st.SaveOneTimeSecret("c1", "the-transient-pw"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f := &fakeKeySvc{}
|
||||
h.SetOffsiteKeyService(f)
|
||||
post := func(path, token, body string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest(http.MethodPost, path, strings.NewReader(body))
|
||||
if token != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
}
|
||||
rr := httptest.NewRecorder()
|
||||
h.ServeHTTP(rr, req)
|
||||
return rr
|
||||
}
|
||||
reg := `{"public_key":"` + testPub + `"}`
|
||||
if rr := post("/api/v1/offsite/register-key/c1", "ckey2", reg); rr.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("cross-customer register → %d, want 401", rr.Code)
|
||||
}
|
||||
if rr := post("/api/v1/offsite/register-key/c1", "ckey", `{"public_key":"command=\"x\" `+testPub+`"}`); rr.Code != http.StatusBadRequest {
|
||||
t.Fatalf("a key with options → %d, want 400 (the hub writes the options)", rr.Code)
|
||||
}
|
||||
for _, c := range []struct{ path, body string }{
|
||||
{"/api/v1/offsite/register-key/c1", reg},
|
||||
{"/api/v1/offsite/confirm-key/c1", `{"fingerprint":"SHA256:fake"}`},
|
||||
{"/api/v1/offsite/move-aside/c1", ``},
|
||||
{"/api/v1/offsite/window-open/c1", `{"count_before":3}`},
|
||||
{"/api/v1/offsite/window-close/c1", `{"window_id":1,"count_after":3,"outcome":"nothing"}`},
|
||||
} {
|
||||
rr := post(c.path, "ckey", c.body)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("%s → %d (%s)", c.path, rr.Code, rr.Body.String())
|
||||
}
|
||||
if strings.Contains(rr.Body.String(), "the-transient-pw") {
|
||||
t.Fatalf("%s leaked the password: %s", c.path, rr.Body.String())
|
||||
}
|
||||
var m map[string]any
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &m); err != nil {
|
||||
t.Fatalf("%s: not JSON: %v", c.path, err)
|
||||
}
|
||||
if _, ok := m["password"]; ok {
|
||||
t.Fatalf("%s: a password field in the response", c.path)
|
||||
}
|
||||
}
|
||||
if f.gotPub != testPub || f.gotFP != "SHA256:fake" {
|
||||
t.Fatalf("service not reached: pub=%q fp=%q", f.gotPub, f.gotFP)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -668,6 +668,16 @@ var operatorOnlyEvents = map[string]bool{
|
||||
// the snapshots still hold the data and telling a customer "your backups were deleted"
|
||||
// would be wrong on the usual reading.
|
||||
"offsite_snapshots_dropped": true,
|
||||
// v0.127.0 (decisions 68–69, R-820/R-822). The off-site key registrar, its daily check and the
|
||||
// clean-up window: custody facts about key lines and fingerprints — the household can take no
|
||||
// action on any of them. Listed in the SAME commit that mints them.
|
||||
"offsite_key_installed": true,
|
||||
"offsite_key_unlocked": true,
|
||||
"offsite_key_audit_failed": true,
|
||||
"offsite_repo_moved_aside": true,
|
||||
"offsite_window_drop": true,
|
||||
"offsite_window_failed": true,
|
||||
"offsite_prune_guard_refused": true,
|
||||
// R-197 (v0.93.0). "The sealed offsite repository key changed" is a custody fact about escrow
|
||||
// blobs. A customer can take no action on it — the remedy is the operator's inspection of the
|
||||
// off-site tier — and the text is operator-grade English naming host ids and retained-blob
|
||||
|
||||
@@ -252,10 +252,13 @@ func (r *Reconciler) heal(ctx context.Context, customerID string, reportID int64
|
||||
return
|
||||
}
|
||||
if restaged {
|
||||
r.logger.Printf("[INFO] offsiteheal: re-staged the stored one-time offsite secret for customer %s (declared %s across %d reports) — the box re-consumes on its next cycle; no provider credential was minted",
|
||||
// v0.127.0 (decision 69): the box no longer receives the password. Re-arming only resets the
|
||||
// delivery ledger; the box gets access by registering its PUBLIC key, which the hub installs with
|
||||
// the credential it keeps sealed. Nothing here reaches the box.
|
||||
r.logger.Printf("[INFO] offsiteheal: customer %s declared %s across %d reports — the stored credential is present (sealed); the box re-registers its public key and the hub installs it; no provider credential was minted",
|
||||
customerID, state, r.debounceReports)
|
||||
r.event(customerID, eventRestaged, "info",
|
||||
"Offsite self-heal: a rebuilt box asked for its storage credential and the stored one-time password was re-armed. No new credential was created at the storage provider.")
|
||||
"Offsite self-heal: a rebuilt box asked for off-site access; the hub still holds the storage credential (sealed) and will install the box's key when it registers. No new credential was created at the storage provider, and none was sent to the box.")
|
||||
r.resetAfterHeal(customerID, reportID, state)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -0,0 +1,386 @@
|
||||
// Package offsitekeys is the hub's off-site KEY REGISTRAR (decision 69, R-820): the hub — never the box —
|
||||
// writes the box's public key into the customer's Storage Box sub-account `.ssh/authorized_keys`, pinned
|
||||
// to an append-only rclone server, and audits that file.
|
||||
//
|
||||
// WHY THE HUB AND NOT THE BOX (measured 2026-10-03, audits/offsite-append-only-2026-10-03/): a key pinned
|
||||
// to `command="rclone serve restic --stdio --append-only <repo>",restrict` can back up and restore and is
|
||||
// refused every delete (403). But the sub-account PASSWORD logs in on ports 22 and 23 and can rewrite
|
||||
// `authorized_keys` — removing the pin. So the protection holds only if no box ever holds the password.
|
||||
// The box sends its PUBLIC key; the hub, which keeps the password sealed (R-821), does the write.
|
||||
//
|
||||
// THE TRANSPORT IS THE PROVIDER'S RESTRICTED SHELL (port 23), not SFTP: measured on the provider, its
|
||||
// `dd of=<file>` takes stdin, `mv` overwrites, `cat` of a missing file exits 1, and `test` does not exist.
|
||||
// That needs only golang.org/x/crypto/ssh, which the hub already uses — no new dependency.
|
||||
//
|
||||
// THE WINDOW (decision 68, Part E): OpenSSH uses the FIRST line whose key matches. Measured on the
|
||||
// provider: the same key on an append-only line first and a deleting line second → refused (403);
|
||||
// deleting line first → deletes. So opening a window = PREPENDING an unpinned-delete line for the box's
|
||||
// own key; closing = removing it. One key on the box.
|
||||
package offsitekeys
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/crypto/ssh"
|
||||
)
|
||||
|
||||
// Shell runs one command in the sub-account's restricted shell, feeding stdin, returning stdout. A
|
||||
// non-zero exit is an error (*ssh.ExitError underneath).
|
||||
type Shell interface {
|
||||
Run(ctx context.Context, cmd string, stdin []byte) ([]byte, error)
|
||||
Close() error
|
||||
}
|
||||
|
||||
// Target is one sub-account, from the customer's offsite descriptor.
|
||||
type Target struct {
|
||||
Host string
|
||||
User string
|
||||
Port int
|
||||
RepoPath string // e.g. /home/felhom-repo — measured: an absolute path works in the forced command
|
||||
Fingerprint string // the host-key fingerprint captured at provisioning (SHA256:…); REQUIRED
|
||||
}
|
||||
|
||||
// Dialer opens a Shell to a Target with the sub-account password, verifying the host key against
|
||||
// Target.Fingerprint (never TOFU).
|
||||
type Dialer interface {
|
||||
Dial(ctx context.Context, t Target, password string) (Shell, error)
|
||||
}
|
||||
|
||||
// PinnedPrefix is the authorized_keys option set every box key must carry outside a window.
|
||||
func PinnedPrefix(repoPath string) string {
|
||||
return fmt.Sprintf(`command="rclone serve restic --stdio --append-only %s",restrict `, repoPath)
|
||||
}
|
||||
|
||||
// WindowPrefix is the DELETING line written only while a clean-up window is open (decision 68).
|
||||
func WindowPrefix(repoPath string) string {
|
||||
return fmt.Sprintf(`command="rclone serve restic --stdio %s",restrict `, repoPath)
|
||||
}
|
||||
|
||||
const authorizedKeys = ".ssh/authorized_keys"
|
||||
const tmpKeys = ".ssh/authorized_keys.felhom-new"
|
||||
|
||||
// Line is one parsed authorized_keys line.
|
||||
type Line struct {
|
||||
Raw string
|
||||
Fingerprint string // SHA256:… of the key, "" when the line holds no parseable key
|
||||
Pinned bool // carries exactly PinnedPrefix(repo)
|
||||
Window bool // carries exactly WindowPrefix(repo)
|
||||
}
|
||||
|
||||
// ParseLines classifies every non-empty, non-comment line of an authorized_keys file.
|
||||
func ParseLines(content, repoPath string) []Line {
|
||||
var out []Line
|
||||
pin, win := PinnedPrefix(repoPath), WindowPrefix(repoPath)
|
||||
for _, raw := range strings.Split(content, "\n") {
|
||||
raw = strings.TrimRight(raw, "\r")
|
||||
t := strings.TrimSpace(raw)
|
||||
if t == "" || strings.HasPrefix(t, "#") {
|
||||
continue
|
||||
}
|
||||
l := Line{Raw: t, Pinned: strings.HasPrefix(t, pin), Window: strings.HasPrefix(t, win)}
|
||||
if pk, _, _, _, err := ssh.ParseAuthorizedKey([]byte(t)); err == nil {
|
||||
l.Fingerprint = ssh.FingerprintSHA256(pk)
|
||||
}
|
||||
out = append(out, l)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// KeyFingerprint validates a single PUBLIC key line as a box would send it (no options) and returns its
|
||||
// fingerprint and its canonical "type base64" form (the comment is dropped — it is box-supplied text).
|
||||
func KeyFingerprint(pub string) (fp, canonical string, err error) {
|
||||
pub = strings.TrimSpace(pub)
|
||||
if pub == "" || strings.ContainsAny(pub, "\n\r\x00") {
|
||||
return "", "", errors.New("offsitekeys: public key must be one line")
|
||||
}
|
||||
pk, _, options, _, err := ssh.ParseAuthorizedKey([]byte(pub))
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("offsitekeys: not a public key: %w", err)
|
||||
}
|
||||
if len(options) > 0 {
|
||||
return "", "", errors.New("offsitekeys: a box key must carry no options — the hub writes them")
|
||||
}
|
||||
switch pk.Type() {
|
||||
case ssh.KeyAlgoED25519, ssh.KeyAlgoRSA, ssh.KeyAlgoECDSA256, ssh.KeyAlgoECDSA384, ssh.KeyAlgoECDSA521:
|
||||
default:
|
||||
return "", "", fmt.Errorf("offsitekeys: key type %s not accepted", pk.Type())
|
||||
}
|
||||
canon := strings.TrimSpace(string(ssh.MarshalAuthorizedKey(pk)))
|
||||
return ssh.FingerprintSHA256(pk), canon, nil
|
||||
}
|
||||
|
||||
// Registrar performs the key-file operations. Every method opens its own Shell and closes it.
|
||||
type Registrar struct {
|
||||
Dialer Dialer
|
||||
}
|
||||
|
||||
func (r *Registrar) open(ctx context.Context, t Target, password string) (Shell, error) {
|
||||
if t.Fingerprint == "" {
|
||||
return nil, errors.New("offsitekeys: target has no host fingerprint — refusing (no blind TOFU)")
|
||||
}
|
||||
if t.Host == "" || t.User == "" || t.RepoPath == "" {
|
||||
return nil, errors.New("offsitekeys: target missing host/user/repo_path")
|
||||
}
|
||||
return r.Dialer.Dial(ctx, t, password)
|
||||
}
|
||||
|
||||
// read returns the current authorized_keys content; a missing file is "" (cat exits 1 there).
|
||||
func read(ctx context.Context, sh Shell) (string, error) {
|
||||
if _, err := sh.Run(ctx, "ls -d .ssh", nil); err != nil {
|
||||
if _, merr := sh.Run(ctx, "mkdir .ssh", nil); merr != nil {
|
||||
return "", fmt.Errorf("offsitekeys: create .ssh: %w", merr)
|
||||
}
|
||||
_, _ = sh.Run(ctx, "chmod 700 .ssh", nil)
|
||||
return "", nil
|
||||
}
|
||||
if _, err := sh.Run(ctx, "ls "+authorizedKeys, nil); err != nil {
|
||||
return "", nil // no file yet
|
||||
}
|
||||
out, err := sh.Run(ctx, "cat "+authorizedKeys, nil)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("offsitekeys: read authorized_keys: %w", err)
|
||||
}
|
||||
return string(out), nil
|
||||
}
|
||||
|
||||
// write replaces authorized_keys atomically (dd to a temp file, chmod, mv) and reads it back.
|
||||
func write(ctx context.Context, sh Shell, content string) error {
|
||||
if _, err := sh.Run(ctx, "dd of="+tmpKeys, []byte(content)); err != nil {
|
||||
return fmt.Errorf("offsitekeys: write temp file: %w", err)
|
||||
}
|
||||
if _, err := sh.Run(ctx, "chmod 600 "+tmpKeys, nil); err != nil {
|
||||
return fmt.Errorf("offsitekeys: chmod temp file: %w", err)
|
||||
}
|
||||
if _, err := sh.Run(ctx, "mv "+tmpKeys+" "+authorizedKeys, nil); err != nil {
|
||||
return fmt.Errorf("offsitekeys: move into place: %w", err)
|
||||
}
|
||||
back, err := sh.Run(ctx, "cat "+authorizedKeys, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("offsitekeys: read back: %w", err)
|
||||
}
|
||||
if strings.TrimSpace(string(back)) != strings.TrimSpace(content) {
|
||||
return errors.New("offsitekeys: read-back differs from what was written")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func join(lines []string) string {
|
||||
if len(lines) == 0 {
|
||||
return ""
|
||||
}
|
||||
return strings.Join(lines, "\n") + "\n"
|
||||
}
|
||||
|
||||
// InstallResult says what Install changed.
|
||||
type InstallResult struct {
|
||||
Fingerprint string
|
||||
RemovedUnpinned int // unpinned lines dropped (any line without the pin is a deletion route)
|
||||
}
|
||||
|
||||
// Install adds the box's key pinned append-only. Every UNPINNED line is dropped in the same write — an
|
||||
// unpinned line is a route to deletion (this is also the migration of a box whose key predates the pin:
|
||||
// the same key comes back pinned). Other PINNED lines are kept until Confirm, so a box that has not yet
|
||||
// switched keeps working (rotation: write new → box confirms → remove old). Idempotent.
|
||||
func (r *Registrar) Install(ctx context.Context, t Target, password, pub string) (InstallResult, error) {
|
||||
fp, canon, err := KeyFingerprint(pub)
|
||||
if err != nil {
|
||||
return InstallResult{}, err
|
||||
}
|
||||
sh, err := r.open(ctx, t, password)
|
||||
if err != nil {
|
||||
return InstallResult{}, err
|
||||
}
|
||||
defer sh.Close()
|
||||
cur, err := read(ctx, sh)
|
||||
if err != nil {
|
||||
return InstallResult{}, err
|
||||
}
|
||||
res := InstallResult{Fingerprint: fp}
|
||||
var keep []string
|
||||
for _, l := range ParseLines(cur, t.RepoPath) {
|
||||
switch {
|
||||
case !l.Pinned:
|
||||
res.RemovedUnpinned++
|
||||
case l.Fingerprint == fp:
|
||||
// re-added below, once
|
||||
default:
|
||||
keep = append(keep, l.Raw)
|
||||
}
|
||||
}
|
||||
keep = append(keep, PinnedPrefix(t.RepoPath)+canon+" felhom-box")
|
||||
if err := write(ctx, sh, join(keep)); err != nil {
|
||||
return InstallResult{}, err
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// Confirm keeps ONLY the pinned line of the confirmed key — the rotation's last step. Returns how many
|
||||
// lines it removed. Refuses when the confirmed key is not present pinned (nothing is written).
|
||||
func (r *Registrar) Confirm(ctx context.Context, t Target, password, fp string) (int, error) {
|
||||
sh, err := r.open(ctx, t, password)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer sh.Close()
|
||||
cur, err := read(ctx, sh)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var keep []string
|
||||
removed := 0
|
||||
for _, l := range ParseLines(cur, t.RepoPath) {
|
||||
if l.Pinned && l.Fingerprint == fp {
|
||||
if len(keep) == 0 {
|
||||
keep = append(keep, l.Raw)
|
||||
} else {
|
||||
removed++
|
||||
}
|
||||
continue
|
||||
}
|
||||
removed++
|
||||
}
|
||||
if len(keep) == 0 {
|
||||
return 0, fmt.Errorf("offsitekeys: key %s is not installed pinned — nothing confirmed", fp)
|
||||
}
|
||||
if removed == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
if err := write(ctx, sh, join(keep)); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return removed, nil
|
||||
}
|
||||
|
||||
// Finding is one problem the audit saw. It names the line by fingerprint, NEVER by key material.
|
||||
type Finding struct {
|
||||
Fingerprint string
|
||||
Kind string // "unpinned" | "window" | "unparseable"
|
||||
}
|
||||
|
||||
// AuditResult is the daily check's view of one sub-account.
|
||||
type AuditResult struct {
|
||||
Lines int
|
||||
Pinned int
|
||||
Findings []Finding
|
||||
}
|
||||
|
||||
// Audit reads authorized_keys and reports every line that is not pinned append-only. A window line is
|
||||
// reported only when no window is supposed to be open. Read-only.
|
||||
func (r *Registrar) Audit(ctx context.Context, t Target, password string, windowOpen bool) (AuditResult, error) {
|
||||
sh, err := r.open(ctx, t, password)
|
||||
if err != nil {
|
||||
return AuditResult{}, err
|
||||
}
|
||||
defer sh.Close()
|
||||
cur, err := read(ctx, sh)
|
||||
if err != nil {
|
||||
return AuditResult{}, err
|
||||
}
|
||||
return audit(cur, t.RepoPath, windowOpen), nil
|
||||
}
|
||||
|
||||
func audit(content, repo string, windowOpen bool) AuditResult {
|
||||
var res AuditResult
|
||||
for _, l := range ParseLines(content, repo) {
|
||||
res.Lines++
|
||||
switch {
|
||||
case l.Pinned:
|
||||
res.Pinned++
|
||||
case l.Window && windowOpen:
|
||||
// expected while the window is open
|
||||
case l.Window:
|
||||
res.Findings = append(res.Findings, Finding{Fingerprint: l.Fingerprint, Kind: "window"})
|
||||
case l.Fingerprint == "":
|
||||
res.Findings = append(res.Findings, Finding{Kind: "unparseable"})
|
||||
default:
|
||||
res.Findings = append(res.Findings, Finding{Fingerprint: l.Fingerprint, Kind: "unpinned"})
|
||||
}
|
||||
}
|
||||
sort.Slice(res.Findings, func(i, j int) bool { return res.Findings[i].Fingerprint < res.Findings[j].Fingerprint })
|
||||
return res
|
||||
}
|
||||
|
||||
// OpenWindow PREPENDS a deleting line for the (pinned, installed) key fp — first match wins (measured).
|
||||
// Refuses when fp is not installed pinned. Idempotent.
|
||||
func (r *Registrar) OpenWindow(ctx context.Context, t Target, password, fp string) error {
|
||||
sh, err := r.open(ctx, t, password)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer sh.Close()
|
||||
cur, err := read(ctx, sh)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var body string
|
||||
var rest []string
|
||||
for _, l := range ParseLines(cur, t.RepoPath) {
|
||||
if l.Window {
|
||||
continue // re-written below exactly once
|
||||
}
|
||||
if l.Pinned && l.Fingerprint == fp && body == "" {
|
||||
body = strings.TrimPrefix(l.Raw, PinnedPrefix(t.RepoPath))
|
||||
}
|
||||
rest = append(rest, l.Raw)
|
||||
}
|
||||
if body == "" {
|
||||
return fmt.Errorf("offsitekeys: key %s is not installed pinned — no window opened", fp)
|
||||
}
|
||||
return write(ctx, sh, join(append([]string{WindowPrefix(t.RepoPath) + body}, rest...)))
|
||||
}
|
||||
|
||||
// CloseWindow removes every window line. Idempotent; a file with no window line is not rewritten.
|
||||
func (r *Registrar) CloseWindow(ctx context.Context, t Target, password string) error {
|
||||
sh, err := r.open(ctx, t, password)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer sh.Close()
|
||||
cur, err := read(ctx, sh)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var rest []string
|
||||
found := false
|
||||
for _, l := range ParseLines(cur, t.RepoPath) {
|
||||
if l.Window {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
rest = append(rest, l.Raw)
|
||||
}
|
||||
if !found {
|
||||
return nil
|
||||
}
|
||||
return write(ctx, sh, join(rest))
|
||||
}
|
||||
|
||||
// MoveAside renames the repository directory to `<repo>.orphaned-<date>` (then -2, -3 …), NEVER
|
||||
// deletes — the R-26/R-32 move-aside the box can no longer do itself (its key reaches only the pinned
|
||||
// rclone server). Returns the new path. A missing repository is an error (nothing to set aside).
|
||||
func (r *Registrar) MoveAside(ctx context.Context, t Target, password, date string) (string, error) {
|
||||
sh, err := r.open(ctx, t, password)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer sh.Close()
|
||||
if _, err := sh.Run(ctx, "ls -d "+t.RepoPath, nil); err != nil {
|
||||
return "", fmt.Errorf("offsitekeys: repository %s not found: %w", t.RepoPath, err)
|
||||
}
|
||||
base := t.RepoPath + ".orphaned-" + date
|
||||
name := base
|
||||
for i := 2; i <= 50; i++ {
|
||||
if _, err := sh.Run(ctx, "ls -d "+name, nil); err != nil {
|
||||
break // absent → free
|
||||
}
|
||||
name = fmt.Sprintf("%s-%d", base, i)
|
||||
}
|
||||
if _, err := sh.Run(ctx, "mv "+t.RepoPath+" "+name, nil); err != nil {
|
||||
return "", fmt.Errorf("offsitekeys: move aside: %w", err)
|
||||
}
|
||||
return name, nil
|
||||
}
|
||||
@@ -0,0 +1,216 @@
|
||||
package offsitekeys
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/ed25519"
|
||||
"crypto/rand"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"golang.org/x/crypto/ssh"
|
||||
)
|
||||
|
||||
// fakeFS emulates the provider's restricted shell as MEASURED 2026-10-03: `dd of=` takes stdin, `mv`
|
||||
// overwrites, `cat`/`ls` of a missing path exit non-zero, `test` does not exist.
|
||||
type fakeFS struct {
|
||||
files map[string]string
|
||||
dirs map[string]bool
|
||||
cmds []string
|
||||
}
|
||||
|
||||
func newFS() *fakeFS { return &fakeFS{files: map[string]string{}, dirs: map[string]bool{".ssh": true}} }
|
||||
|
||||
func (f *fakeFS) Run(_ context.Context, cmd string, stdin []byte) ([]byte, error) {
|
||||
f.cmds = append(f.cmds, cmd)
|
||||
a := strings.Fields(cmd)
|
||||
miss := errors.New("exit status 1")
|
||||
switch {
|
||||
case a[0] == "ls" && a[1] == "-d":
|
||||
if f.dirs[a[2]] {
|
||||
return []byte(a[2] + "\n"), nil
|
||||
}
|
||||
return nil, miss
|
||||
case a[0] == "ls":
|
||||
if _, ok := f.files[a[1]]; ok {
|
||||
return []byte(a[1]), nil
|
||||
}
|
||||
return nil, miss
|
||||
case a[0] == "cat":
|
||||
if v, ok := f.files[a[1]]; ok {
|
||||
return []byte(v), nil
|
||||
}
|
||||
return nil, miss
|
||||
case a[0] == "mkdir":
|
||||
f.dirs[a[1]] = true
|
||||
return nil, nil
|
||||
case a[0] == "chmod":
|
||||
return nil, nil
|
||||
case strings.HasPrefix(a[0], "dd") && strings.HasPrefix(a[1], "of="):
|
||||
f.files[strings.TrimPrefix(a[1], "of=")] = string(stdin)
|
||||
return nil, nil
|
||||
case a[0] == "mv":
|
||||
if v, ok := f.files[a[1]]; ok {
|
||||
f.files[a[2]] = v
|
||||
delete(f.files, a[1])
|
||||
return nil, nil
|
||||
}
|
||||
if f.dirs[a[1]] {
|
||||
f.dirs[a[2]] = true
|
||||
delete(f.dirs, a[1])
|
||||
return nil, nil
|
||||
}
|
||||
return nil, miss
|
||||
case a[0] == "rm":
|
||||
return nil, errors.New("the registrar must never delete")
|
||||
}
|
||||
return nil, errors.New("Command not found")
|
||||
}
|
||||
func (f *fakeFS) Close() error { return nil }
|
||||
|
||||
type fakeDialer struct{ fs *fakeFS }
|
||||
|
||||
func (d fakeDialer) Dial(context.Context, Target, string) (Shell, error) { return d.fs, nil }
|
||||
|
||||
var tgt = Target{Host: "u1-sub4.example", User: "u1-sub4", Port: 23, RepoPath: "/home/felhom-repo", Fingerprint: "SHA256:host"}
|
||||
|
||||
func newKey(t *testing.T) (pub, fp string) {
|
||||
t.Helper()
|
||||
k, _, err := ed25519.GenerateKey(rand.Reader)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pk, _ := ssh.NewPublicKey(k)
|
||||
return strings.TrimSpace(string(ssh.MarshalAuthorizedKey(pk))) + " box", ssh.FingerprintSHA256(pk)
|
||||
}
|
||||
|
||||
// The MIGRATION shape: a box whose key predates the pin (an unpinned line, exactly as ssh-copy-id left
|
||||
// it) registers the SAME key → it comes back pinned and the unpinned line is gone. Then the audit is clean.
|
||||
func TestInstall_MigratesUnpinnedKeyAndAuditGoesClean(t *testing.T) {
|
||||
fs := newFS()
|
||||
pub, fp := newKey(t)
|
||||
other, _ := newKey(t)
|
||||
fs.files[".ssh/authorized_keys"] = pub + "\n" + other + "\n"
|
||||
r := &Registrar{Dialer: fakeDialer{fs}}
|
||||
|
||||
before, err := r.Audit(context.Background(), tgt, "pw", false)
|
||||
if err != nil || len(before.Findings) != 2 {
|
||||
t.Fatalf("before: %+v %v — want 2 unpinned findings (the decoy must be seen)", before, err)
|
||||
}
|
||||
res, err := r.Install(context.Background(), tgt, "pw", pub)
|
||||
if err != nil || res.Fingerprint != fp || res.RemovedUnpinned != 2 {
|
||||
t.Fatalf("install = %+v, %v", res, err)
|
||||
}
|
||||
got := fs.files[".ssh/authorized_keys"]
|
||||
if !strings.HasPrefix(got, PinnedPrefix(tgt.RepoPath)) || strings.Count(got, "\n") != 1 {
|
||||
t.Fatalf("file after install:\n%s", got)
|
||||
}
|
||||
after, _ := r.Audit(context.Background(), tgt, "pw", false)
|
||||
if len(after.Findings) != 0 || after.Pinned != 1 {
|
||||
t.Fatalf("after: %+v", after)
|
||||
}
|
||||
for _, c := range fs.cmds {
|
||||
if strings.HasPrefix(c, "rm") {
|
||||
t.Fatalf("registrar issued a delete: %q", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Rotation: new key installed beside the old pinned one; Confirm leaves only the new one.
|
||||
func TestInstallThenConfirm_Rotation(t *testing.T) {
|
||||
fs := newFS()
|
||||
r := &Registrar{Dialer: fakeDialer{fs}}
|
||||
oldPub, oldFP := newKey(t)
|
||||
newPub, newFP := newKey(t)
|
||||
if _, err := r.Install(context.Background(), tgt, "pw", oldPub); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := r.Install(context.Background(), tgt, "pw", newPub); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lines := ParseLines(fs.files[".ssh/authorized_keys"], tgt.RepoPath)
|
||||
if len(lines) != 2 || !lines[0].Pinned || !lines[1].Pinned {
|
||||
t.Fatalf("both keys must be pinned until confirm: %+v", lines)
|
||||
}
|
||||
if _, err := r.Confirm(context.Background(), tgt, "pw", "SHA256:not-installed"); err == nil {
|
||||
t.Fatal("confirming an absent key must refuse")
|
||||
}
|
||||
n, err := r.Confirm(context.Background(), tgt, "pw", newFP)
|
||||
if err != nil || n != 1 {
|
||||
t.Fatalf("confirm = %d, %v", n, err)
|
||||
}
|
||||
lines = ParseLines(fs.files[".ssh/authorized_keys"], tgt.RepoPath)
|
||||
if len(lines) != 1 || lines[0].Fingerprint != newFP || lines[0].Fingerprint == oldFP {
|
||||
t.Fatalf("after confirm: %+v", lines)
|
||||
}
|
||||
}
|
||||
|
||||
// The window (decision 68): the deleting line goes FIRST (first match wins — measured), the audit
|
||||
// tolerates it only while a window is open, and closing removes it.
|
||||
func TestWindow_PrependAuditClose(t *testing.T) {
|
||||
fs := newFS()
|
||||
r := &Registrar{Dialer: fakeDialer{fs}}
|
||||
pub, fp := newKey(t)
|
||||
if err := r.OpenWindow(context.Background(), tgt, "pw", fp); err == nil {
|
||||
t.Fatal("a window for a key that is not installed must refuse")
|
||||
}
|
||||
if _, err := r.Install(context.Background(), tgt, "pw", pub); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := r.OpenWindow(context.Background(), tgt, "pw", fp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lines := ParseLines(fs.files[".ssh/authorized_keys"], tgt.RepoPath)
|
||||
if len(lines) != 2 || !lines[0].Window || !lines[1].Pinned || lines[0].Fingerprint != fp {
|
||||
t.Fatalf("window line must be first: %+v", lines)
|
||||
}
|
||||
if a, _ := r.Audit(context.Background(), tgt, "pw", true); len(a.Findings) != 0 {
|
||||
t.Fatalf("open window flagged: %+v", a)
|
||||
}
|
||||
if a, _ := r.Audit(context.Background(), tgt, "pw", false); len(a.Findings) != 1 || a.Findings[0].Kind != "window" {
|
||||
t.Fatalf("a window line with no open window must alarm: %+v", a)
|
||||
}
|
||||
if err := r.CloseWindow(context.Background(), tgt, "pw"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if a, _ := r.Audit(context.Background(), tgt, "pw", false); len(a.Findings) != 0 || a.Pinned != 1 {
|
||||
t.Fatalf("after close: %+v", a)
|
||||
}
|
||||
}
|
||||
|
||||
// Move-aside renames, never deletes, and never reuses a name.
|
||||
func TestMoveAside_RenamesNeverDeletes(t *testing.T) {
|
||||
fs := newFS()
|
||||
fs.dirs["/home/felhom-repo"] = true
|
||||
fs.dirs["/home/felhom-repo.orphaned-20261003"] = true
|
||||
r := &Registrar{Dialer: fakeDialer{fs}}
|
||||
to, err := r.MoveAside(context.Background(), tgt, "pw", "20261003")
|
||||
if err != nil || to != "/home/felhom-repo.orphaned-20261003-2" {
|
||||
t.Fatalf("move-aside = %q, %v", to, err)
|
||||
}
|
||||
if fs.dirs["/home/felhom-repo"] || !fs.dirs["/home/felhom-repo.orphaned-20261003"] {
|
||||
t.Fatalf("dirs after: %v", fs.dirs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTargetWithoutFingerprint_Refused(t *testing.T) {
|
||||
r := &Registrar{Dialer: fakeDialer{newFS()}}
|
||||
pub, _ := newKey(t)
|
||||
nt := tgt
|
||||
nt.Fingerprint = ""
|
||||
if _, err := r.Install(context.Background(), nt, "pw", pub); err == nil {
|
||||
t.Fatal("no host fingerprint must refuse (no blind TOFU)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestKeyFingerprint_RefusesOptionsAndJunk(t *testing.T) {
|
||||
pub, _ := newKey(t)
|
||||
for _, bad := range []string{"", "not a key", `command="sh" ` + pub, pub + "\n" + pub} {
|
||||
if _, _, err := KeyFingerprint(bad); err == nil {
|
||||
t.Fatalf("accepted %q", bad)
|
||||
}
|
||||
}
|
||||
if _, _, err := KeyFingerprint(pub); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,335 @@
|
||||
package offsitekeys
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// Event types this package raises — all OPERATOR-ONLY (registered in notify.operatorOnlyEvents).
|
||||
const (
|
||||
EventKeyInstalled = "offsite_key_installed" // info: the registrar pinned a box key
|
||||
EventKeyUnlocked = "offsite_key_unlocked" // error: the daily check saw a line that can delete
|
||||
EventAuditFailed = "offsite_key_audit_failed" // warning: the daily check could not read the file
|
||||
EventMovedAside = "offsite_repo_moved_aside" // info: the hub set an orphaned repository aside
|
||||
)
|
||||
|
||||
// Service binds the Registrar to the hub's store: the descriptor (where), the sealed password (how),
|
||||
// the key record, and the operator events.
|
||||
type Service struct {
|
||||
Store *store.Store
|
||||
Reg *Registrar
|
||||
Logger *log.Logger
|
||||
// Emit routes an event to the dispatcher (operator mail). nil → events are only saved.
|
||||
Emit func(customerID, eventType, severity, message, detailsJSON, source string)
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
// ErrNotProvisioned — the customer has no provisioned off-site target (nothing to register against).
|
||||
var ErrNotProvisioned = errors.New("offsitekeys: customer has no provisioned off-site target")
|
||||
|
||||
func (s *Service) logf(f string, a ...any) {
|
||||
if s.Logger != nil {
|
||||
s.Logger.Printf(f, a...)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Service) now() time.Time {
|
||||
if s.Now != nil {
|
||||
return s.Now()
|
||||
}
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
func (s *Service) event(customerID, typ, sev, msg string, details any) {
|
||||
dj := ""
|
||||
if details != nil {
|
||||
if b, err := json.Marshal(details); err == nil {
|
||||
dj = string(b)
|
||||
}
|
||||
}
|
||||
if _, err := s.Store.SaveEvent(customerID, typ, sev, msg, dj, "hub"); err != nil {
|
||||
s.logf("[WARN] offsitekeys: save event %s for %s: %v", typ, customerID, err)
|
||||
}
|
||||
if s.Emit != nil {
|
||||
s.Emit(customerID, typ, sev, msg, dj, "hub")
|
||||
}
|
||||
}
|
||||
|
||||
// TargetFor resolves the customer's sub-account and the hub's (decrypted) password for it.
|
||||
func (s *Service) TargetFor(customerID string) (Target, string, error) {
|
||||
cfg, err := s.Store.GetCustomerConfig(customerID)
|
||||
if err != nil || cfg == nil {
|
||||
return Target{}, "", fmt.Errorf("offsitekeys: customer %s: %v", customerID, err)
|
||||
}
|
||||
d, err := offsite.ReadDescriptor(cfg.ConfigJSON)
|
||||
if err != nil {
|
||||
return Target{}, "", err
|
||||
}
|
||||
if d == nil || !d.Enabled || d.Host == "" || d.User == "" || d.RepoPath == "" {
|
||||
return Target{}, "", ErrNotProvisioned
|
||||
}
|
||||
pw, err := s.Store.OffsitePassword(customerID)
|
||||
if err != nil {
|
||||
return Target{}, "", fmt.Errorf("offsitekeys: no usable stored credential for %s: %w", customerID, err)
|
||||
}
|
||||
return Target{Host: d.Host, User: d.User, Port: d.Port, RepoPath: d.RepoPath, Fingerprint: d.HostFingerprint}, pw, nil
|
||||
}
|
||||
|
||||
// RegisterKey installs the box's public key pinned append-only (the ONLY way a box gets off-site access
|
||||
// from hub v0.127.0 on — no box ever receives the password).
|
||||
func (s *Service) RegisterKey(ctx context.Context, customerID, pub string) (InstallResult, error) {
|
||||
t, pw, err := s.TargetFor(customerID)
|
||||
if err != nil {
|
||||
return InstallResult{}, err
|
||||
}
|
||||
start := s.now()
|
||||
res, err := s.Reg.Install(ctx, t, pw, pub)
|
||||
if err != nil {
|
||||
s.logf("[ERROR] offsitekeys: install key for %s (%s@%s): %v", customerID, t.User, t.Host, err)
|
||||
return InstallResult{}, err
|
||||
}
|
||||
if err := s.Store.RecordOffsiteKeyInstalled(customerID, res.Fingerprint); err != nil {
|
||||
s.logf("[WARN] offsitekeys: record key for %s: %v", customerID, err)
|
||||
}
|
||||
if err := s.Store.MarkOffsiteSecretDelivered(customerID); err != nil {
|
||||
s.logf("[WARN] offsitekeys: mark delivered for %s: %v", customerID, err)
|
||||
}
|
||||
s.logf("[INFO] offsitekeys: installed box key %s for %s pinned append-only (%s@%s, dropped %d unpinned line(s)) in %s",
|
||||
res.Fingerprint, customerID, t.User, t.Host, res.RemovedUnpinned, s.now().Sub(start).Round(time.Millisecond))
|
||||
s.event(customerID, EventKeyInstalled, "info",
|
||||
fmt.Sprintf("Off-site: the box's key %s was installed append-only on %s (%d unpinned line(s) removed).", res.Fingerprint, t.User, res.RemovedUnpinned),
|
||||
map[string]any{"fingerprint": res.Fingerprint, "removed_unpinned": res.RemovedUnpinned})
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// ConfirmKey is the rotation's last step: only the confirmed key's pinned line stays.
|
||||
func (s *Service) ConfirmKey(ctx context.Context, customerID, fp string) (int, error) {
|
||||
t, pw, err := s.TargetFor(customerID)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
removed, err := s.Reg.Confirm(ctx, t, pw, fp)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if ok, err := s.Store.RecordOffsiteKeyConfirmed(customerID, fp); err != nil || !ok {
|
||||
s.logf("[WARN] offsitekeys: confirm record for %s (fp %s): matched=%v err=%v", customerID, fp, ok, err)
|
||||
}
|
||||
s.logf("[INFO] offsitekeys: box confirmed key %s for %s; %d other line(s) removed", fp, customerID, removed)
|
||||
return removed, nil
|
||||
}
|
||||
|
||||
// MoveAside sets the repository aside (never deletes) on the box's request.
|
||||
func (s *Service) MoveAside(ctx context.Context, customerID string) (string, error) {
|
||||
t, pw, err := s.TargetFor(customerID)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
name, err := s.Reg.MoveAside(ctx, t, pw, s.now().UTC().Format("20060102"))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
s.logf("[WARN] offsitekeys: moved %s's repository aside: %s -> %s (nothing deleted)", customerID, t.RepoPath, name)
|
||||
s.event(customerID, EventMovedAside, "info",
|
||||
fmt.Sprintf("Off-site: the box asked to set its orphaned repository aside; %s was moved to %s. Nothing was deleted.", t.RepoPath, name),
|
||||
map[string]any{"from": t.RepoPath, "to": name})
|
||||
return name, nil
|
||||
}
|
||||
|
||||
// AuditOutcome is one customer's daily-check result.
|
||||
type AuditOutcome struct {
|
||||
CustomerID string
|
||||
Result AuditResult
|
||||
Err error
|
||||
}
|
||||
|
||||
// WindowOpenFunc reports whether a clean-up window is open for the customer (Part E). nil → never.
|
||||
type WindowOpenFunc func(customerID string) bool
|
||||
|
||||
// AuditAll is the DAILY CHECK (decision 69): every provisioned customer's authorized_keys is read, and
|
||||
// any line that can delete outside an open window raises `offsite_key_unlocked` (error, operator-only),
|
||||
// naming the line by fingerprint only. An unreadable file raises `offsite_key_audit_failed`.
|
||||
func (s *Service) AuditAll(ctx context.Context, windowOpen WindowOpenFunc) []AuditOutcome {
|
||||
cfgs, err := s.Store.ListCustomerConfigs()
|
||||
if err != nil {
|
||||
s.logf("[ERROR] offsitekeys: audit: list configs: %v", err)
|
||||
return nil
|
||||
}
|
||||
var out []AuditOutcome
|
||||
for _, c := range cfgs {
|
||||
d, derr := offsite.ReadDescriptor(c.ConfigJSON)
|
||||
if derr != nil || d == nil || !d.Enabled || d.Host == "" {
|
||||
continue
|
||||
}
|
||||
o := AuditOutcome{CustomerID: c.CustomerID}
|
||||
t, pw, terr := s.TargetFor(c.CustomerID)
|
||||
if terr != nil {
|
||||
o.Err = terr
|
||||
} else {
|
||||
open := windowOpen != nil && windowOpen(c.CustomerID)
|
||||
o.Result, o.Err = s.Reg.Audit(ctx, t, pw, open)
|
||||
}
|
||||
out = append(out, o)
|
||||
switch {
|
||||
case o.Err != nil:
|
||||
s.logf("[WARN] offsitekeys: audit %s: %v", c.CustomerID, o.Err)
|
||||
s.event(c.CustomerID, EventAuditFailed, "warning",
|
||||
fmt.Sprintf("Off-site key check: could not read the key file of %s: %v", c.CustomerID, o.Err), nil)
|
||||
case len(o.Result.Findings) > 0:
|
||||
var parts []string
|
||||
for _, f := range o.Result.Findings {
|
||||
parts = append(parts, f.Kind+" "+f.Fingerprint)
|
||||
}
|
||||
s.logf("[ERROR] offsitekeys: audit %s: %d line(s) can delete off-site history: %s", c.CustomerID, len(o.Result.Findings), strings.Join(parts, "; "))
|
||||
s.event(c.CustomerID, EventKeyUnlocked, "error",
|
||||
fmt.Sprintf("Off-site key check: %d key line(s) on %s are NOT append-only and can delete this household's off-site history: %s",
|
||||
len(o.Result.Findings), t.User, strings.Join(parts, "; ")),
|
||||
map[string]any{"findings": o.Result.Findings, "lines": o.Result.Lines, "pinned": o.Result.Pinned})
|
||||
default:
|
||||
s.logf("[INFO] offsitekeys: audit %s: %d line(s), all pinned append-only", c.CustomerID, o.Result.Lines)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ── Decision 68: the clean-up window ──────────────────────────────────────────────────────────────
|
||||
|
||||
const (
|
||||
EventWindowDrop = "offsite_window_drop" // error: more snapshots went than a window may remove
|
||||
EventWindowFailed = "offsite_window_failed" // warning: a window errored or was left open
|
||||
EventGuardRefused = "offsite_prune_guard_refused" // error: the box's fake-snapshot guard refused (R-822)
|
||||
windowLength = 20 * time.Minute
|
||||
windowCadence = 6*24*time.Hour + 12*time.Hour // "weekly", with slack for the night chain's drift
|
||||
)
|
||||
|
||||
// WindowGrant is the hub's answer to the box.
|
||||
type WindowGrant struct {
|
||||
Granted bool `json:"granted"`
|
||||
WindowID int64 `json:"window_id,omitempty"`
|
||||
NewestAllowed time.Time `json:"newest_allowed,omitempty"`
|
||||
MaxRemove int `json:"max_remove,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
// WindowResult is the box's report when it is done.
|
||||
type WindowResult struct {
|
||||
WindowID int64 `json:"window_id"`
|
||||
CountBefore int `json:"count_before"`
|
||||
CountAfter int `json:"count_after"`
|
||||
Removed int `json:"removed"`
|
||||
Outcome string `json:"outcome"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
// MaxRemove is the most snapshots one window may remove: 40 % of what was there, at least 5. The ruled
|
||||
// policy (7 daily + 4 weekly + 6 monthly per app) removes ~7 of ~17 per app per week (~41 %); the box
|
||||
// takes the OLDEST first within this bound, so a backlog drains over several windows.
|
||||
func MaxRemove(countBefore int) int {
|
||||
n := countBefore * 40 / 100
|
||||
if n < 5 {
|
||||
n = 5
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// OpenWindowFor decides and, if due, opens the window: a deleting line for the box's CONFIRMED key is
|
||||
// prepended (first match wins), and a ledger row bounds it to 20 minutes.
|
||||
func (s *Service) OpenWindowFor(ctx context.Context, customerID string, countBefore int) (WindowGrant, error) {
|
||||
oneShot := s.Store.TakeOffsiteWindowGrant(customerID)
|
||||
last := s.Store.LastOffsiteWindowOpened(customerID)
|
||||
due := last.IsZero() || s.now().Sub(last) >= windowCadence
|
||||
if !oneShot && !(s.Store.OffsiteWindowsEnabled() && due) {
|
||||
reason := "weekly windows are off"
|
||||
if s.Store.OffsiteWindowsEnabled() {
|
||||
reason = "not due (last window " + last.UTC().Format(time.RFC3339) + ")"
|
||||
}
|
||||
return WindowGrant{Reason: reason}, nil
|
||||
}
|
||||
k, err := s.Store.GetOffsiteKey(customerID)
|
||||
if err != nil || k == nil || k.ConfirmedAt.IsZero() {
|
||||
return WindowGrant{Reason: "no confirmed append-only key on record"}, nil
|
||||
}
|
||||
t, pw, err := s.TargetFor(customerID)
|
||||
if err != nil {
|
||||
return WindowGrant{}, err
|
||||
}
|
||||
if err := s.Reg.OpenWindow(ctx, t, pw, k.Fingerprint); err != nil {
|
||||
return WindowGrant{}, err
|
||||
}
|
||||
now := s.now()
|
||||
id, err := s.Store.OpenOffsiteWindowRow(customerID, now.Add(windowLength), countBefore)
|
||||
if err != nil {
|
||||
// The line is written; close it rather than leave a deleting line without a ledger row.
|
||||
_ = s.Reg.CloseWindow(ctx, t, pw)
|
||||
return WindowGrant{}, err
|
||||
}
|
||||
g := WindowGrant{Granted: true, WindowID: id, NewestAllowed: now.UTC(), MaxRemove: MaxRemove(countBefore)}
|
||||
s.logf("[WARN] offsitekeys: clean-up window %d OPENED for %s (key %s, %d snapshot(s), max %d removed, closes by %s, one-shot=%v)",
|
||||
id, customerID, k.Fingerprint, countBefore, g.MaxRemove, now.Add(windowLength).UTC().Format(time.RFC3339), oneShot)
|
||||
return g, nil
|
||||
}
|
||||
|
||||
// CloseWindowFor closes the window on the box's report and checks the count.
|
||||
func (s *Service) CloseWindowFor(ctx context.Context, customerID string, r WindowResult) error {
|
||||
w, err := s.Store.GetOffsiteWindow(r.WindowID)
|
||||
if err != nil || w == nil || w.CustomerID != customerID {
|
||||
return fmt.Errorf("offsitekeys: window %d is not this customer's", r.WindowID)
|
||||
}
|
||||
t, pw, err := s.TargetFor(customerID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.Reg.CloseWindow(ctx, t, pw); err != nil {
|
||||
return err // the sweep retries at closes_by
|
||||
}
|
||||
if _, err := s.Store.CloseOffsiteWindowRow(w.ID, r.CountAfter, r.Outcome, "box"); err != nil {
|
||||
s.logf("[WARN] offsitekeys: ledger close %d: %v", w.ID, err)
|
||||
}
|
||||
drop := w.CountBefore - r.CountAfter
|
||||
s.logf("[INFO] offsitekeys: clean-up window %d CLOSED for %s: outcome=%s, %d -> %d (drop %d, allowed %d)",
|
||||
w.ID, customerID, r.Outcome, w.CountBefore, r.CountAfter, drop, MaxRemove(w.CountBefore))
|
||||
details := map[string]any{"window_id": w.ID, "count_before": w.CountBefore, "count_after": r.CountAfter, "outcome": r.Outcome, "reason": r.Reason}
|
||||
switch {
|
||||
case drop > MaxRemove(w.CountBefore):
|
||||
s.event(customerID, EventWindowDrop, "error",
|
||||
fmt.Sprintf("Off-site clean-up window %d: the snapshot count fell %d -> %d, more than a window may remove (%d).", w.ID, w.CountBefore, r.CountAfter, MaxRemove(w.CountBefore)), details)
|
||||
case r.Outcome == "guard-refused":
|
||||
s.event(customerID, EventGuardRefused, "error",
|
||||
fmt.Sprintf("Off-site clean-up window %d: the box's fake-snapshot guard refused to prune — nothing was deleted: %s", w.ID, r.Reason), details)
|
||||
case r.Outcome == "error":
|
||||
s.event(customerID, EventWindowFailed, "warning",
|
||||
fmt.Sprintf("Off-site clean-up window %d: the prune failed on the box: %s", w.ID, r.Reason), details)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SweepExpiredWindows closes every window left open past its 20 minutes (a box that crashed or lied).
|
||||
func (s *Service) SweepExpiredWindows(ctx context.Context) {
|
||||
ws, err := s.Store.ExpiredOffsiteWindows()
|
||||
if err != nil {
|
||||
s.logf("[WARN] offsitekeys: window sweep: %v", err)
|
||||
return
|
||||
}
|
||||
for _, w := range ws {
|
||||
t, pw, err := s.TargetFor(w.CustomerID)
|
||||
if err == nil {
|
||||
err = s.Reg.CloseWindow(ctx, t, pw)
|
||||
}
|
||||
if err != nil {
|
||||
s.logf("[ERROR] offsitekeys: window %d for %s is past its time and could NOT be closed: %v (retrying next sweep)", w.ID, w.CustomerID, err)
|
||||
continue
|
||||
}
|
||||
_, _ = s.Store.CloseOffsiteWindowRow(w.ID, -1, "", "timeout")
|
||||
s.logf("[WARN] offsitekeys: clean-up window %d for %s was left open — closed by the hub", w.ID, w.CustomerID)
|
||||
s.event(w.CustomerID, EventWindowFailed, "warning",
|
||||
fmt.Sprintf("Off-site clean-up window %d was not closed by the box within %s; the hub closed it (the deleting key line is removed).", w.ID, windowLength), nil)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
package offsitekeys
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
func svcFixture(t *testing.T) (*Service, *fakeFS, *[]string) {
|
||||
t.Helper()
|
||||
st, err := store.New(filepath.Join(t.TempDir(), "hub.db"), log.New(os.Stderr, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { st.Close() })
|
||||
cfg := `{"offsite":{"enabled":true,"type":"shared","host":"u1-sub4.example","user":"u1-sub4","port":23,"repo_path":"/home/felhom-repo","host_fingerprint":"SHA256:host"}}`
|
||||
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "k", RetrievalPassword: "p", ConfigJSON: cfg}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveOneTimeSecret("c1", "SubPw1%"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fs := newFS()
|
||||
var events []string
|
||||
s := &Service{Store: st, Reg: &Registrar{Dialer: fakeDialer{fs}},
|
||||
Emit: func(_, typ, _, _, _, _ string) { events = append(events, typ) }}
|
||||
return s, fs, &events
|
||||
}
|
||||
|
||||
// Register → confirm → the window is refused while weekly windows are off; an operator one-shot opens
|
||||
// it (deleting line FIRST); the box's close removes it; a drop beyond the bound alarms.
|
||||
func TestWindow_GrantOpenCloseAndDropAlarm(t *testing.T) {
|
||||
s, fs, events := svcFixture(t)
|
||||
ctx := context.Background()
|
||||
pub, fp := newKey(t)
|
||||
if _, err := s.RegisterKey(ctx, "c1", pub); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.ConfirmKey(ctx, "c1", fp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if g, err := s.OpenWindowFor(ctx, "c1", 20); err != nil || g.Granted {
|
||||
t.Fatalf("windows off: granted=%v err=%v — must refuse", g.Granted, err)
|
||||
}
|
||||
if err := s.Store.GrantOffsiteWindowOnce("c1"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
g, err := s.OpenWindowFor(ctx, "c1", 20)
|
||||
if err != nil || !g.Granted || g.MaxRemove != 8 {
|
||||
t.Fatalf("one-shot: %+v %v", g, err)
|
||||
}
|
||||
if lines := ParseLines(fs.files[".ssh/authorized_keys"], "/home/felhom-repo"); !lines[0].Window || lines[0].Fingerprint != fp {
|
||||
t.Fatalf("window line not first: %+v", lines)
|
||||
}
|
||||
if !s.Store.OffsiteWindowOpen("c1") {
|
||||
t.Fatal("the ledger does not show the window open — the daily check would alarm on it")
|
||||
}
|
||||
if g2, _ := s.OpenWindowFor(ctx, "c1", 20); g2.Granted {
|
||||
t.Fatal("the one-shot grant was not consumed")
|
||||
}
|
||||
// The box reports a fall of 12 (allowed 8) → offsite_window_drop.
|
||||
if err := s.CloseWindowFor(ctx, "c1", WindowResult{WindowID: g.WindowID, CountAfter: 8, Outcome: "pruned"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if a := audit(fs.files[".ssh/authorized_keys"], "/home/felhom-repo", false); len(a.Findings) != 0 {
|
||||
t.Fatalf("window line left behind: %+v", a)
|
||||
}
|
||||
found := false
|
||||
for _, e := range *events {
|
||||
if e == EventWindowDrop {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("no %s event for a fall beyond the bound: %v", EventWindowDrop, *events)
|
||||
}
|
||||
}
|
||||
|
||||
// A window for a box that never confirmed its key is refused (nothing to scope the window to).
|
||||
func TestWindow_NoConfirmedKeyRefused(t *testing.T) {
|
||||
s, _, _ := svcFixture(t)
|
||||
_ = s.Store.GrantOffsiteWindowOnce("c1")
|
||||
pub, _ := newKey(t)
|
||||
if _, err := s.RegisterKey(context.Background(), "c1", pub); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if g, _ := s.OpenWindowFor(context.Background(), "c1", 10); g.Granted {
|
||||
t.Fatal("granted without a confirmed key")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package offsitekeys
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"golang.org/x/crypto/ssh"
|
||||
)
|
||||
|
||||
// SSHDialer is the production Dialer: password (and keyboard-interactive) auth to the sub-account's
|
||||
// restricted shell, the host key checked against the provisioning-time fingerprint. The password is
|
||||
// never logged and never leaves this process.
|
||||
type SSHDialer struct {
|
||||
Timeout time.Duration // connect + per-command bound; 0 → 30 s
|
||||
}
|
||||
|
||||
type sshShell struct {
|
||||
c *ssh.Client
|
||||
timeout time.Duration
|
||||
}
|
||||
|
||||
func (d SSHDialer) Dial(ctx context.Context, t Target, password string) (Shell, error) {
|
||||
to := d.Timeout
|
||||
if to == 0 {
|
||||
to = 30 * time.Second
|
||||
}
|
||||
port := t.Port
|
||||
if port == 0 {
|
||||
port = 23
|
||||
}
|
||||
want := t.Fingerprint
|
||||
cfg := &ssh.ClientConfig{
|
||||
User: t.User,
|
||||
Auth: []ssh.AuthMethod{
|
||||
ssh.Password(password),
|
||||
ssh.KeyboardInteractive(func(_, _ string, qs []string, _ []bool) ([]string, error) {
|
||||
ans := make([]string, len(qs))
|
||||
for i := range ans {
|
||||
ans[i] = password
|
||||
}
|
||||
return ans, nil
|
||||
}),
|
||||
},
|
||||
HostKeyCallback: func(_ string, _ net.Addr, key ssh.PublicKey) error {
|
||||
if got := ssh.FingerprintSHA256(key); got != want {
|
||||
return fmt.Errorf("offsitekeys: host key MISMATCH for %s (got %s, want %s)", t.Host, got, want)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
Timeout: to,
|
||||
}
|
||||
addr := net.JoinHostPort(t.Host, strconv.Itoa(port))
|
||||
var nd net.Dialer
|
||||
dctx, cancel := context.WithTimeout(ctx, to)
|
||||
defer cancel()
|
||||
conn, err := nd.DialContext(dctx, "tcp", addr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("offsitekeys: dial %s: %w", addr, err)
|
||||
}
|
||||
_ = conn.SetDeadline(time.Now().Add(to))
|
||||
cc, chans, reqs, err := ssh.NewClientConn(conn, addr, cfg)
|
||||
if err != nil {
|
||||
conn.Close()
|
||||
return nil, fmt.Errorf("offsitekeys: ssh handshake %s: %w", addr, err)
|
||||
}
|
||||
_ = conn.SetDeadline(time.Time{})
|
||||
return &sshShell{c: ssh.NewClient(cc, chans, reqs), timeout: to}, nil
|
||||
}
|
||||
|
||||
func (s *sshShell) Run(ctx context.Context, cmd string, stdin []byte) ([]byte, error) {
|
||||
sess, err := s.c.NewSession()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer sess.Close()
|
||||
if stdin != nil {
|
||||
sess.Stdin = bytes.NewReader(stdin)
|
||||
}
|
||||
var out, errb bytes.Buffer
|
||||
sess.Stdout, sess.Stderr = &out, &errb
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- sess.Run(cmd) }()
|
||||
rctx, cancel := context.WithTimeout(ctx, s.timeout)
|
||||
defer cancel()
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
return out.Bytes(), fmt.Errorf("%q: %w: %s", cmd, err, bytes.TrimSpace(errb.Bytes()))
|
||||
}
|
||||
return out.Bytes(), nil
|
||||
case <-rctx.Done():
|
||||
_ = sess.Close()
|
||||
return nil, fmt.Errorf("%q: %w", cmd, rctx.Err())
|
||||
}
|
||||
}
|
||||
|
||||
func (s *sshShell) Close() error { return s.c.Close() }
|
||||
@@ -0,0 +1,177 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"time"
|
||||
)
|
||||
|
||||
// OffsiteKey is the registrar's record of the box key the hub installed pinned (decision 69).
|
||||
type OffsiteKey struct {
|
||||
CustomerID string
|
||||
Fingerprint string
|
||||
InstalledAt time.Time
|
||||
ConfirmedAt time.Time // zero = the box has not confirmed it yet
|
||||
}
|
||||
|
||||
// RecordOffsiteKeyInstalled records (last-write-wins) the key the hub just installed; confirmation resets.
|
||||
func (s *Store) RecordOffsiteKeyInstalled(customerID, fp string) error {
|
||||
_, err := s.db.Exec(`
|
||||
INSERT INTO offsite_keys (customer_id, fingerprint, installed_at, confirmed_at) VALUES (?, ?, datetime('now'), NULL)
|
||||
ON CONFLICT(customer_id) DO UPDATE SET fingerprint = excluded.fingerprint, installed_at = datetime('now'), confirmed_at = NULL`,
|
||||
customerID, fp)
|
||||
return err
|
||||
}
|
||||
|
||||
// RecordOffsiteKeyConfirmed marks the installed key confirmed by the box; false when fp is not the key on record.
|
||||
func (s *Store) RecordOffsiteKeyConfirmed(customerID, fp string) (bool, error) {
|
||||
res, err := s.db.Exec(`UPDATE offsite_keys SET confirmed_at = datetime('now') WHERE customer_id = ? AND fingerprint = ?`, customerID, fp)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
n, _ := res.RowsAffected()
|
||||
return n > 0, nil
|
||||
}
|
||||
|
||||
// GetOffsiteKey returns the record, or (nil, nil) when none exists.
|
||||
func (s *Store) GetOffsiteKey(customerID string) (*OffsiteKey, error) {
|
||||
var k OffsiteKey
|
||||
var inst string
|
||||
var conf sql.NullString
|
||||
err := s.db.QueryRow(`SELECT customer_id, fingerprint, installed_at, confirmed_at FROM offsite_keys WHERE customer_id = ?`, customerID).
|
||||
Scan(&k.CustomerID, &k.Fingerprint, &inst, &conf)
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
k.InstalledAt = parseSQLiteTime(inst)
|
||||
if conf.Valid {
|
||||
k.ConfirmedAt = parseSQLiteTime(conf.String)
|
||||
}
|
||||
return &k, nil
|
||||
}
|
||||
|
||||
// OffsiteWindowOpen reports whether a clean-up window is open for the customer right now (decision 68):
|
||||
// a window row not closed and not past its closes_by. Errors read as "closed" — the key check then
|
||||
// alarms on a window line, which is the safe side.
|
||||
func (s *Store) OffsiteWindowOpen(customerID string) bool {
|
||||
var n int
|
||||
err := s.db.QueryRow(`SELECT COUNT(*) FROM offsite_windows WHERE customer_id = ? AND closed_at IS NULL AND closes_by > datetime('now')`, customerID).Scan(&n)
|
||||
return err == nil && n > 0
|
||||
}
|
||||
|
||||
// OffsiteWindow is one clean-up window (decision 68).
|
||||
type OffsiteWindow struct {
|
||||
ID int64
|
||||
CustomerID string
|
||||
OpenedAt time.Time
|
||||
ClosesBy time.Time
|
||||
ClosedAt time.Time
|
||||
CountBefore int
|
||||
CountAfter int
|
||||
BoxResult string
|
||||
CloseReason string
|
||||
}
|
||||
|
||||
// OpenOffsiteWindowRow records a window the hub just opened.
|
||||
func (s *Store) OpenOffsiteWindowRow(customerID string, closesBy time.Time, countBefore int) (int64, error) {
|
||||
res, err := s.db.Exec(`INSERT INTO offsite_windows (customer_id, opened_at, closes_by, count_before) VALUES (?, datetime('now'), ?, ?)`,
|
||||
customerID, closesBy.UTC().Format("2006-01-02 15:04:05"), countBefore)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return res.LastInsertId()
|
||||
}
|
||||
|
||||
// CloseOffsiteWindowRow closes a window (idempotent: an already-closed row is not touched).
|
||||
func (s *Store) CloseOffsiteWindowRow(id int64, countAfter int, boxResult, reason string) (bool, error) {
|
||||
res, err := s.db.Exec(`UPDATE offsite_windows SET closed_at = datetime('now'), count_after = ?, box_result = ?, close_reason = ? WHERE id = ? AND closed_at IS NULL`,
|
||||
countAfter, boxResult, reason, id)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
n, _ := res.RowsAffected()
|
||||
return n > 0, nil
|
||||
}
|
||||
|
||||
// GetOffsiteWindow returns one window row, or (nil, nil).
|
||||
func (s *Store) GetOffsiteWindow(id int64) (*OffsiteWindow, error) {
|
||||
var w OffsiteWindow
|
||||
var opened, closesBy string
|
||||
var closed, boxRes, reason sql.NullString
|
||||
var before, after sql.NullInt64
|
||||
err := s.db.QueryRow(`SELECT id, customer_id, opened_at, closes_by, closed_at, count_before, count_after, box_result, close_reason FROM offsite_windows WHERE id = ?`, id).
|
||||
Scan(&w.ID, &w.CustomerID, &opened, &closesBy, &closed, &before, &after, &boxRes, &reason)
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
w.OpenedAt, w.ClosesBy = parseSQLiteTime(opened), parseSQLiteTime(closesBy)
|
||||
if closed.Valid {
|
||||
w.ClosedAt = parseSQLiteTime(closed.String)
|
||||
}
|
||||
w.CountBefore, w.CountAfter = int(before.Int64), int(after.Int64)
|
||||
w.BoxResult, w.CloseReason = boxRes.String, reason.String
|
||||
return &w, nil
|
||||
}
|
||||
|
||||
// LastOffsiteWindowOpened returns when the customer's most recent window was opened (zero = never).
|
||||
func (s *Store) LastOffsiteWindowOpened(customerID string) time.Time {
|
||||
var v sql.NullString
|
||||
if err := s.db.QueryRow(`SELECT MAX(opened_at) FROM offsite_windows WHERE customer_id = ?`, customerID).Scan(&v); err != nil || !v.Valid {
|
||||
return time.Time{}
|
||||
}
|
||||
return parseSQLiteTime(v.String)
|
||||
}
|
||||
|
||||
// ExpiredOffsiteWindows lists windows still open past their closes_by.
|
||||
func (s *Store) ExpiredOffsiteWindows() ([]OffsiteWindow, error) {
|
||||
rows, err := s.db.Query(`SELECT id, customer_id FROM offsite_windows WHERE closed_at IS NULL AND closes_by <= datetime('now')`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []OffsiteWindow
|
||||
for rows.Next() {
|
||||
var w OffsiteWindow
|
||||
if err := rows.Scan(&w.ID, &w.CustomerID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, w)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
const offsiteWindowsEnabledKey = "offsite_prune_windows_enabled"
|
||||
|
||||
// OffsiteWindowsEnabled — the operator switch for WEEKLY windows (decision 68). Off by default: the
|
||||
// interim is "nothing prunes" until the operator turns the weekly window on.
|
||||
func (s *Store) OffsiteWindowsEnabled() bool { return s.getSetting(offsiteWindowsEnabledKey) == "on" }
|
||||
|
||||
// SetOffsiteWindowsEnabled flips the weekly switch.
|
||||
func (s *Store) SetOffsiteWindowsEnabled(on bool) error {
|
||||
v := ""
|
||||
if on {
|
||||
v = "on"
|
||||
}
|
||||
return s.setSetting(offsiteWindowsEnabledKey, v)
|
||||
}
|
||||
|
||||
// GrantOffsiteWindowOnce lets the customer's NEXT window request through regardless of the weekly
|
||||
// cadence (operator one-shot).
|
||||
func (s *Store) GrantOffsiteWindowOnce(customerID string) error {
|
||||
return s.setSetting("offsite_window_grant:"+customerID, "1")
|
||||
}
|
||||
|
||||
// TakeOffsiteWindowGrant consumes a one-shot grant; true when one was present.
|
||||
func (s *Store) TakeOffsiteWindowGrant(customerID string) bool {
|
||||
k := "offsite_window_grant:" + customerID
|
||||
if s.getSetting(k) != "1" {
|
||||
return false
|
||||
}
|
||||
_ = s.setSetting(k, "")
|
||||
return true
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"crypto/aes"
|
||||
"crypto/cipher"
|
||||
"crypto/rand"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ── R-821 / decision 69: the off-site sub-account password is stored ENCRYPTED, with a key that is not
|
||||
// in the database ─────────────────────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Until hub v0.127.0 `one_time_secrets.value` held every customer's Storage Box sub-account password in
|
||||
// the clear, forever (the value survives a consume on purpose — RestageOneTimeSecret). Measured
|
||||
// 2026-10-03: the stored value for tester-1 still logged in to its sub-account, and that password can
|
||||
// rewrite `.ssh/authorized_keys` — i.e. remove the append-only pin from any box's key (R-820). So a copy
|
||||
// of hub.db alone was enough to erase every household's off-site history.
|
||||
//
|
||||
// Now: AES-256-GCM, a fresh nonce per write, stored as `enc:v1:<base64(nonce||ciphertext)>`. The key
|
||||
// comes from the hub's environment (OFFSITE_SECRET_KEY, from the out-of-band k8s Secret — never the
|
||||
// database, never git). Without a key the store REFUSES to save (fail-closed): a hub that cannot seal
|
||||
// must not quietly fall back to plaintext. A row that is still plaintext from before the upgrade is
|
||||
// sealed in place by SealLegacyOffsiteSecrets at start-up.
|
||||
//
|
||||
// Pinned by: TestOffsiteSecret_RawRowHoldsNoPassword, TestOffsiteSecret_WrongKeyCannotOpen,
|
||||
// TestOffsiteSecret_NoKeyRefusesToSave, TestSealLegacyOffsiteSecrets_SealsPlaintextRows.
|
||||
|
||||
const sealPrefix = "enc:v1:"
|
||||
|
||||
// ErrNoSealKey is returned when an off-site secret is saved or read on a store with no sealing key.
|
||||
var ErrNoSealKey = errors.New("store: no off-site secret sealing key configured (OFFSITE_SECRET_KEY)")
|
||||
|
||||
// SetOffsiteSecretKey installs the 32-byte AES-256 key used to seal off-site sub-account passwords.
|
||||
func (s *Store) SetOffsiteSecretKey(key []byte) error {
|
||||
if len(key) != 32 {
|
||||
return fmt.Errorf("store: off-site secret key must be 32 bytes, got %d", len(key))
|
||||
}
|
||||
blk, err := aes.NewCipher(key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
gcm, err := cipher.NewGCM(blk)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.sealer = gcm
|
||||
return nil
|
||||
}
|
||||
|
||||
// ParseOffsiteSecretKey decodes OFFSITE_SECRET_KEY: 64 hex characters or standard base64 of 32 bytes.
|
||||
func ParseOffsiteSecretKey(v string) ([]byte, error) {
|
||||
v = strings.TrimSpace(v)
|
||||
if len(v) == 64 {
|
||||
if b, err := hex.DecodeString(v); err == nil {
|
||||
return b, nil
|
||||
}
|
||||
}
|
||||
if b, err := base64.StdEncoding.DecodeString(v); err == nil && len(b) == 32 {
|
||||
return b, nil
|
||||
}
|
||||
return nil, errors.New("OFFSITE_SECRET_KEY must be 64 hex characters or base64 of 32 bytes")
|
||||
}
|
||||
|
||||
func (s *Store) sealSecret(plain string) (string, error) {
|
||||
if s.sealer == nil {
|
||||
return "", ErrNoSealKey
|
||||
}
|
||||
nonce := make([]byte, s.sealer.NonceSize())
|
||||
if _, err := rand.Read(nonce); err != nil {
|
||||
return "", err
|
||||
}
|
||||
ct := s.sealer.Seal(nil, nonce, []byte(plain), nil)
|
||||
return sealPrefix + base64.StdEncoding.EncodeToString(append(nonce, ct...)), nil
|
||||
}
|
||||
|
||||
func (s *Store) openSecret(stored string) (string, error) {
|
||||
if s.sealer == nil {
|
||||
return "", ErrNoSealKey
|
||||
}
|
||||
if !strings.HasPrefix(stored, sealPrefix) {
|
||||
return "", errors.New("store: off-site secret is not sealed (run SealLegacyOffsiteSecrets)")
|
||||
}
|
||||
raw, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(stored, sealPrefix))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("store: sealed secret: %w", err)
|
||||
}
|
||||
ns := s.sealer.NonceSize()
|
||||
if len(raw) < ns {
|
||||
return "", errors.New("store: sealed secret too short")
|
||||
}
|
||||
pt, err := s.sealer.Open(nil, raw[:ns], raw[ns:], nil)
|
||||
if err != nil {
|
||||
return "", errors.New("store: sealed secret does not open with this key")
|
||||
}
|
||||
return string(pt), nil
|
||||
}
|
||||
|
||||
// OffsitePassword returns the customer's sub-account password, decrypted, for the HUB's own use (the key
|
||||
// registrar, decision 69). It does NOT mark anything consumed and it is never served to a box.
|
||||
// sql.ErrNoRows when none is stored.
|
||||
func (s *Store) OffsitePassword(customerID string) (string, error) {
|
||||
var v string
|
||||
if err := s.db.QueryRow(`SELECT value FROM one_time_secrets WHERE customer_id = ?`, customerID).Scan(&v); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return s.openSecret(v)
|
||||
}
|
||||
|
||||
// MarkOffsiteSecretDelivered records that the stored credential has been USED to deliver a key to the
|
||||
// box (the registrar installed it). It keeps the delivery-state machinery (R-70) meaningful now that no
|
||||
// box consumes the password: consumed_at = "delivered", exactly as before, without the value leaving.
|
||||
func (s *Store) MarkOffsiteSecretDelivered(customerID string) error {
|
||||
_, err := s.db.Exec(`UPDATE one_time_secrets SET consumed_at = datetime('now') WHERE customer_id = ?`, customerID)
|
||||
return err
|
||||
}
|
||||
|
||||
// SealLegacyOffsiteSecrets seals, in place, every row still holding a plaintext value (written before
|
||||
// v0.127.0). Idempotent. Returns how many rows it sealed. Values are never logged.
|
||||
func (s *Store) SealLegacyOffsiteSecrets() (int, error) {
|
||||
if s.sealer == nil {
|
||||
return 0, ErrNoSealKey
|
||||
}
|
||||
rows, err := s.db.Query(`SELECT customer_id, value FROM one_time_secrets`)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
type row struct{ id, v string }
|
||||
var todo []row
|
||||
for rows.Next() {
|
||||
var r row
|
||||
if err := rows.Scan(&r.id, &r.v); err != nil {
|
||||
rows.Close()
|
||||
return 0, err
|
||||
}
|
||||
if !strings.HasPrefix(r.v, sealPrefix) {
|
||||
todo = append(todo, r)
|
||||
}
|
||||
}
|
||||
rows.Close()
|
||||
n := 0
|
||||
for _, r := range todo {
|
||||
sealed, err := s.sealSecret(r.v)
|
||||
if err != nil {
|
||||
return n, err
|
||||
}
|
||||
if _, err := s.db.Exec(`UPDATE one_time_secrets SET value = ? WHERE customer_id = ? AND value = ?`, sealed, r.id, r.v); err != nil {
|
||||
return n, err
|
||||
}
|
||||
n++
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func sealTestStore(t *testing.T) *Store {
|
||||
t.Helper()
|
||||
st, err := New(filepath.Join(t.TempDir(), "hub.db"), log.New(os.Stderr, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { st.Close() })
|
||||
return st
|
||||
}
|
||||
|
||||
func rawValue(t *testing.T, st *Store, id string) string {
|
||||
t.Helper()
|
||||
var v string
|
||||
if err := st.db.QueryRow(`SELECT value FROM one_time_secrets WHERE customer_id = ?`, id).Scan(&v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// R-821: a copy of the database alone does not reveal the password — asserted on the raw column.
|
||||
func TestOffsiteSecret_RawRowHoldsNoPassword(t *testing.T) {
|
||||
st := sealTestStore(t)
|
||||
if err := st.SaveOneTimeSecret("c1", "Sup3rSecretPw%"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw := rawValue(t, st, "c1")
|
||||
if strings.Contains(raw, "Sup3rSecretPw") || !strings.HasPrefix(raw, sealPrefix) {
|
||||
t.Fatalf("raw row is not sealed: %q", raw)
|
||||
}
|
||||
if pw, err := st.OffsitePassword("c1"); err != nil || pw != "Sup3rSecretPw%" {
|
||||
t.Fatalf("OffsitePassword = %q, %v", pw, err)
|
||||
}
|
||||
if pw, err := st.ConsumeOneTimeSecret("c1"); err != nil || pw != "Sup3rSecretPw%" {
|
||||
t.Fatalf("Consume = %q, %v", pw, err)
|
||||
}
|
||||
}
|
||||
|
||||
// A different key cannot open what was sealed (the key is the secret, not the format).
|
||||
func TestOffsiteSecret_WrongKeyCannotOpen(t *testing.T) {
|
||||
st := sealTestStore(t)
|
||||
if err := st.SaveOneTimeSecret("c1", "Sup3rSecretPw%"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SetOffsiteSecretKey([]byte("another-key-of-exactly-32-bytes!")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pw, err := st.OffsitePassword("c1"); err == nil || pw != "" {
|
||||
t.Fatalf("wrong key opened the secret: %q", pw)
|
||||
}
|
||||
}
|
||||
|
||||
// Fail-closed: no key → nothing saved, nothing read, never plaintext.
|
||||
func TestOffsiteSecret_NoKeyRefusesToSave(t *testing.T) {
|
||||
st := sealTestStore(t)
|
||||
st.sealer = nil
|
||||
if err := st.SaveOneTimeSecret("c1", "Sup3rSecretPw%"); err != ErrNoSealKey {
|
||||
t.Fatalf("save without key = %v, want ErrNoSealKey", err)
|
||||
}
|
||||
var n int
|
||||
_ = st.db.QueryRow(`SELECT COUNT(*) FROM one_time_secrets`).Scan(&n)
|
||||
if n != 0 {
|
||||
t.Fatalf("%d row(s) written without a key", n)
|
||||
}
|
||||
}
|
||||
|
||||
// Rows written by a pre-v0.127.0 hub are sealed in place at start-up; idempotent.
|
||||
func TestSealLegacyOffsiteSecrets_SealsPlaintextRows(t *testing.T) {
|
||||
st := sealTestStore(t)
|
||||
if _, err := st.db.Exec(`INSERT INTO one_time_secrets (customer_id, value) VALUES ('old', 'LegacyPlain1%')`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveOneTimeSecret("new", "AlreadySealed1%"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sealedNew := rawValue(t, st, "new")
|
||||
n, err := st.SealLegacyOffsiteSecrets()
|
||||
if err != nil || n != 1 {
|
||||
t.Fatalf("sealed %d, %v; want 1", n, err)
|
||||
}
|
||||
if raw := rawValue(t, st, "old"); strings.Contains(raw, "LegacyPlain") {
|
||||
t.Fatalf("legacy row still plaintext: %q", raw)
|
||||
}
|
||||
if rawValue(t, st, "new") != sealedNew {
|
||||
t.Fatal("an already-sealed row was re-sealed")
|
||||
}
|
||||
if pw, err := st.OffsitePassword("old"); err != nil || pw != "LegacyPlain1%" {
|
||||
t.Fatalf("legacy row does not open: %q %v", pw, err)
|
||||
}
|
||||
if n, _ := st.SealLegacyOffsiteSecrets(); n != 0 {
|
||||
t.Fatalf("second run sealed %d", n)
|
||||
}
|
||||
if _, err := st.OffsitePassword("absent"); err != sql.ErrNoRows {
|
||||
t.Fatalf("absent = %v, want ErrNoRows", err)
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"crypto/cipher"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
@@ -8,6 +9,7 @@ import (
|
||||
"log"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/semver"
|
||||
@@ -24,6 +26,10 @@ type Store struct {
|
||||
// defaultMinControllerVersion is the config/env-supplied global FLOOR fallback (Phase 2 managed
|
||||
// updates). Used only when neither a per-customer override nor a hub_settings row is set.
|
||||
defaultMinControllerVersion string
|
||||
|
||||
// sealer encrypts the off-site sub-account password at rest (R-821, decision 69). nil = no key
|
||||
// configured → saving an off-site secret is REFUSED (offsite_seal.go).
|
||||
sealer cipher.AEAD
|
||||
}
|
||||
|
||||
// SetDefaultMinControllerVersion sets the config/env-supplied global floor fallback. Called once at
|
||||
@@ -97,6 +103,12 @@ func New(dbPath string, logger *log.Logger) (*Store, error) {
|
||||
}
|
||||
|
||||
s := &Store{db: db, logger: logger}
|
||||
// Under `go test` ONLY (testing.Testing() is false in the production binary) every store gets a
|
||||
// fixed sealing key, so the ~40 test files that build a store need no ceremony. Production stays
|
||||
// fail-closed until main installs OFFSITE_SECRET_KEY. TestOffsiteSecret_NoKeyRefusesToSave clears it.
|
||||
if testing.Testing() {
|
||||
_ = s.SetOffsiteSecretKey([]byte("felhom-hub-test-only-seal-key-32"))
|
||||
}
|
||||
if err := s.migrate(); err != nil {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("migrating database: %w", err)
|
||||
@@ -809,6 +821,31 @@ func (s *Store) migrate() error {
|
||||
s.logger.Printf("[INFO] [store] app_stopped_unhealthy added to %d household(s)' notification prefs (one-time, add-only): %v", len(changed), changed)
|
||||
}
|
||||
|
||||
// v0.127.0 (decisions 68–69, R-820): the off-site key registrar's record — which box key the hub
|
||||
// installed pinned append-only, and when the box confirmed it — and the clean-up window ledger.
|
||||
if _, err := s.db.Exec(`
|
||||
CREATE TABLE IF NOT EXISTS offsite_keys (
|
||||
customer_id TEXT PRIMARY KEY,
|
||||
fingerprint TEXT NOT NULL,
|
||||
installed_at DATETIME NOT NULL DEFAULT (datetime('now')),
|
||||
confirmed_at DATETIME
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS offsite_windows (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
customer_id TEXT NOT NULL,
|
||||
opened_at DATETIME NOT NULL DEFAULT (datetime('now')),
|
||||
closes_by DATETIME NOT NULL,
|
||||
closed_at DATETIME,
|
||||
count_before INTEGER,
|
||||
count_after INTEGER,
|
||||
box_result TEXT,
|
||||
close_reason TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_offsite_windows_customer ON offsite_windows(customer_id, opened_at);
|
||||
`); err != nil {
|
||||
return fmt.Errorf("offsite_keys/offsite_windows: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -1586,6 +1623,11 @@ func (s *Store) GetCustomerConfig(customerID string) (*CustomerConfig, error) {
|
||||
// SaveOneTimeSecret stores (last-write-wins) the one-time transient offsite password for a customer,
|
||||
// resetting the consumed flag (a fresh provision supersedes any prior unconsumed value). Never logged.
|
||||
func (s *Store) SaveOneTimeSecret(customerID, value string) error {
|
||||
sealed, serr := s.sealSecret(value) // R-821: never stored in the clear; no key → refuse
|
||||
if serr != nil {
|
||||
return serr
|
||||
}
|
||||
value = sealed
|
||||
_, err := s.db.Exec(`
|
||||
INSERT INTO one_time_secrets (customer_id, value, created_at, consumed_at)
|
||||
VALUES (?, ?, datetime('now'), NULL)
|
||||
@@ -1608,6 +1650,9 @@ func (s *Store) ConsumeOneTimeSecret(customerID string) (string, error) {
|
||||
if err != nil {
|
||||
return "", err // sql.ErrNoRows when absent OR already consumed
|
||||
}
|
||||
if value, err = s.openSecret(value); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if _, err := tx.Exec(`UPDATE one_time_secrets SET consumed_at = datetime('now') WHERE customer_id = ?`, customerID); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
+54
-10
@@ -65,16 +65,21 @@ type Server struct {
|
||||
versionChecker *VersionChecker
|
||||
templateFetcher *TemplateFetcher
|
||||
assetsMgr *assets.Manager
|
||||
gitea *gitea.Client // optional; enables the Day-0 artifact version dropdowns
|
||||
offsite *offsite.Provisioner // optional; enables Hetzner offsite provisioning (SLICE 1)
|
||||
offsiteBox func() (monitor.BoxSnapshot, bool) // optional (v0.64.0, R-5); the restic pool-box aggregate snapshot accessor
|
||||
pbsdrBox func() (monitor.PBSBoxSnapshot, bool) // optional (v0.65.0, R-5); the PBS-DR datastore fill snapshot accessor
|
||||
tenantsync tenancyProvisioner // optional; enables PBS DR tier provisioning (web/pbsdr.go)
|
||||
claimEngine *claim.Engine // optional; enables the customer-claim resend button (v0.50.0)
|
||||
selfBindMailer SelfBindMailer // optional; enables the customer self-bind link button (v0.66.0, R-27)
|
||||
bindLimiter *bindRateLimiter // per-IP throttle for the PUBLIC /bind/ surface (v0.66.0, R-27)
|
||||
bindResendMu sync.Mutex // R-719: the fresh-link resend limiter
|
||||
bindResendAt map[string]time.Time // customer → last fresh-link mail (R-719)
|
||||
gitea *gitea.Client // optional; enables the Day-0 artifact version dropdowns
|
||||
offsite *offsite.Provisioner // optional; enables Hetzner offsite provisioning (SLICE 1)
|
||||
// offsiteKeyAudit runs the daily off-site key check on demand (decision 69). nil → 503.
|
||||
offsiteKeyAudit func(ctx context.Context) any
|
||||
// offsiteWindowAdmin: operator one-shot grant / weekly switch (decision 68). nil → 503.
|
||||
offsiteWindowGrant func(customerID string) error
|
||||
offsiteWindowSwitch func(on bool) error
|
||||
offsiteBox func() (monitor.BoxSnapshot, bool) // optional (v0.64.0, R-5); the restic pool-box aggregate snapshot accessor
|
||||
pbsdrBox func() (monitor.PBSBoxSnapshot, bool) // optional (v0.65.0, R-5); the PBS-DR datastore fill snapshot accessor
|
||||
tenantsync tenancyProvisioner // optional; enables PBS DR tier provisioning (web/pbsdr.go)
|
||||
claimEngine *claim.Engine // optional; enables the customer-claim resend button (v0.50.0)
|
||||
selfBindMailer SelfBindMailer // optional; enables the customer self-bind link button (v0.66.0, R-27)
|
||||
bindLimiter *bindRateLimiter // per-IP throttle for the PUBLIC /bind/ surface (v0.66.0, R-27)
|
||||
bindResendMu sync.Mutex // R-719: the fresh-link resend limiter
|
||||
bindResendAt map[string]time.Time // customer → last fresh-link mail (R-719)
|
||||
// intentHub (v0.58.0, Direction-2 immediate-sync) is Bumped by every operator-intent handler
|
||||
// (config save/delete, claim resend, offsite re-issue/freeze, floor, block/unblock, log pull)
|
||||
// so a box long-polling GET /api/v1/wait wakes in seconds. Shared with the API handler. nil =
|
||||
@@ -194,6 +199,14 @@ func (s *Server) SetAssetManager(am *assets.Manager) {
|
||||
// offsite enabled returns an error (offsite not configured on this hub).
|
||||
func (s *Server) SetOffsiteProvisioner(p *offsite.Provisioner) { s.offsite = p }
|
||||
|
||||
// SetOffsiteKeyAudit wires the on-demand run of the daily off-site key check (decision 69).
|
||||
func (s *Server) SetOffsiteKeyAudit(fn func(ctx context.Context) any) { s.offsiteKeyAudit = fn }
|
||||
|
||||
// SetOffsiteWindowAdmin wires the operator's window controls (decision 68).
|
||||
func (s *Server) SetOffsiteWindowAdmin(grant func(string) error, sw func(bool) error) {
|
||||
s.offsiteWindowGrant, s.offsiteWindowSwitch = grant, sw
|
||||
}
|
||||
|
||||
// SetOffsiteBox wires the pool-box aggregate snapshot accessor (v0.64.0, R-5): the checker's cached
|
||||
// snapshot, read on the Offsite tab + the Dashboard tile. nil (no HETZNER_TOKEN/box id) → both render an
|
||||
// honest "not configured". The web layer NEVER fetches from Hetzner — it only reads this cache.
|
||||
@@ -604,6 +617,37 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
} else {
|
||||
s.handleConfigEditForm(w, r, customerID)
|
||||
}
|
||||
case strings.HasPrefix(path, "/offsite/window-grant/") || path == "/offsite/windows-enabled":
|
||||
// Operator (decision 68): a one-shot grant lets the customer's NEXT window request through;
|
||||
// the switch turns the WEEKLY window on/off fleet-wide (off = the interim: nothing prunes).
|
||||
if r.Method != http.MethodPost || s.offsiteWindowGrant == nil {
|
||||
http.Error(w, "unavailable", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
var err error
|
||||
if path == "/offsite/windows-enabled" {
|
||||
err = s.offsiteWindowSwitch(r.FormValue("on") == "1")
|
||||
} else {
|
||||
err = s.offsiteWindowGrant(strings.TrimPrefix(path, "/offsite/window-grant/"))
|
||||
}
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte("{\"ok\":true}\n"))
|
||||
case path == "/offsite/key-audit":
|
||||
// Operator: run the daily off-site key check now (decision 69). Same code path as the 07:10 job.
|
||||
if r.Method != http.MethodPost {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
if s.offsiteKeyAudit == nil {
|
||||
http.Error(w, "off-site key check not configured", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(s.offsiteKeyAudit(r.Context()))
|
||||
case strings.HasPrefix(path, "/configs/") && strings.HasSuffix(path, "/offsite-reissue"):
|
||||
customerID := strings.TrimPrefix(path, "/configs/")
|
||||
customerID = strings.TrimSuffix(customerID, "/offsite-reissue")
|
||||
|
||||
Reference in New Issue
Block a user