diff --git a/documentation/architecture/04-control-plane-authorization.md b/documentation/architecture/04-control-plane-authorization.md index 041628bb..87b2abe7 100644 --- a/documentation/architecture/04-control-plane-authorization.md +++ b/documentation/architecture/04-control-plane-authorization.md @@ -111,7 +111,9 @@ operational key until the first PAYING customer exists; testers do not count.** per box (the blob binds `host_id`), so one box moves at a time and a fenced box cannot be swept along. Proven twice: `demo-hp-bb76ea` 2026-09-15, `demo-felhom-8363b5` 2026-09-16. **Measured 2026-10-09 (R-924):** all three key files open with an EMPTY passphrase, and `/mnt/5_hdd/felhom.eu` is in no backup — so today neither the "passphrase- -protected" nor the "kept offline" of §1 holds, and a loss of DooPlex loses both keys at once. **Revisit on the first +protected" nor the "kept offline" of §1 holds, and a loss of DooPlex loses both keys at once. **Operator ruling 2026-10-09 (R-924, option A):** both keys now ride the +nightly encrypted DooPlex→ep0 copy (behind the DooPlex off-site key) and go on the printed break-glass sheet; the +recovery key is still not passphrase-protected. **Revisit on the first sale** — at that point the key belongs behind the operator (or a hardware key, §7), and a fleet rollout step still has to be designed (R-530). diff --git a/documentation/audits/dooplex-survival-2026-10-09/signing/red-proof.txt b/documentation/audits/dooplex-survival-2026-10-09/signing/red-proof.txt new file mode 100644 index 00000000..5fab6268 --- /dev/null +++ b/documentation/audits/dooplex-survival-2026-10-09/signing/red-proof.txt @@ -0,0 +1,18 @@ +### RED: missing-key refusal removed +FAIL: test_a_missing_signing_key_refuses (__main__.Push.test_a_missing_signing_key_refuses) +Ran 1 test in 0.540s +FAILED (failures=1) +### RED: restore: key/public-half match removed +FAIL: test_restore_refuses_a_signing_key_that_does_not_match (__main__.RestoreTest.test_restore_refuses_a_signing_key_that_does_not_match) +Ran 1 test in 0.763s +FAILED (failures=1) +### RED: signing step skipped +FAIL: test_signing_keys_ride_the_copy (__main__.Push.test_signing_keys_ride_the_copy) +Ran 1 test in 0.521s +FAILED (failures=1) +### GREEN (GNU) +Ran 30 tests in 43.420s +OK +### GREEN (BusyBox tools + fake sqlite3) +Ran 30 tests in 44.500s +OK diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index ea0034ed..74dd5d97 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -196,7 +196,7 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-924** | Security & access | P2 | **The operator signing keys exist only on DooPlex, with no passphrase and in no backup — a loss of DooPlex means no box ever accepts a signed agent update, config bundle or OS step again without an on-site re-enrollment.** MEASURED 2026-10-09: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` (0600 `kisfenyo`); `ssh-keygen -y -P ''` opens all three (no passphrase); `/mnt/5_hdd/felhom.eu` is in no backup set, the off-site job does not carry them. `04` §1 says the operational key is "passphrase-protected" and the recovery key "kept offline" — today neither holds (custody ruling 1 of 2026-09-16 put both on DooPlex for this phase; `04` §3.1). Both keys share one fate, so the cold key cannot do its one job (re-pin after losing the operational key). `runbooks/total-loss-of-dooplex.md` S6. | **WAITING-ON-OPERATOR** — custody: print both (the break-glass sheet has the lines and commands) and/or add them to the nightly encrypted copy (a DooPlex change), or move the recovery key off DooPlex entirely. | operator decision | Operator: pick; CC: the copy change if picked | operator | +| **R-924** | Security & access | P2 | **The operator signing keys exist only on DooPlex, with no passphrase and in no backup — a loss of DooPlex means no box ever accepts a signed agent update, config bundle or OS step again without an on-site re-enrollment.** MEASURED 2026-10-09: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` (0600 `kisfenyo`); `ssh-keygen -y -P ''` opens all three (no passphrase); `/mnt/5_hdd/felhom.eu` is in no backup set, the off-site job does not carry them. `04` §1 says the operational key is "passphrase-protected" and the recovery key "kept offline" — today neither holds (custody ruling 1 of 2026-09-16 put both on DooPlex for this phase; `04` §3.1). Both keys share one fate, so the cold key cannot do its one job (re-pin after losing the operational key). `runbooks/total-loss-of-dooplex.md` S6. | **NARROWED 2026-10-09 — operator ruling: option A** (print both AND copy them). The three key files and their public halves ride the nightly encrypted DooPlex→ep0 copy (`scripts/dooplex-offsite/`, a missing key refuses the push); the Sunday restore test proves each private key still derives its own public half (never printed). **LEFT:** the operator prints S6a/S6b (`runbooks/break-glass-sheet.md`); the keys still have no passphrase (custody ruling 1 of 2026-09-16 stands until the first sale). | operator: print | Operator: print the two keys; then close | operator | | **R-132** | Security & access | P3 | **`curl -w '%{redirect_url}'` reconstructs the request URL WITH its basic-auth credential** — so a `-u ":$HUB_PW"` call that never put the password in a URL still printed it **Merged 2026-10-05 from R-350 (duplicate):** (1) 2026-08-20 occurrence: POST /configuration/artifacts answers 303; leak lives only in the CC transcript under ~/.claude/projects/ on DooPlex, not in git/evidence (checked then). (2) `-v` and `--libcurl` also re-render the credential, not only %{redirect_url}; confirm redirects with %{http_code} + follow-up GET. (3) Rotation path: hub /configuration form (current_password/new_password/confirm_password); DB override wins over ConfigMap (break-glass); CC can rotate file-to-file without printing (operator-present-one-time-secrets) if asked. | **WAITING-ON-OPERATOR** — **ACTION: rotate `HUB_PW`** **Folded R-580 2026-10-03** (the same `curl -w %{redirect_url}` credential echo, seen again 2026-09-18). | — | Happened on 2026-07-31 while red-proofing the R-120 gate: the hub operator password was written to the session transcript by the write-out format, not by the request. `-u` is safe; the *reporting* was not. Rule: read the redirect from `-D -` and grep `^Location:`, never `%{redirect_url}`, on any authenticated call. Rotate the hub password (`/configuration` → Login password; ConfigMap `auth.password_hash` is the reset path) and update `~/.config/credentials` | Viktor | | **R-137** | Security & access | P3 | **Cloudflare geo-WAF rules are zone-scoped and non-namespaced — four cross-tenant faults.** `globalRuleDesc = "[felhom-geo] Global"` (`waf.go:18`) is one literal description per ZONE; `appRuleDescPrefix` keys by app name with no customer (`waf.go:21`); `BuildGlobalExpression` has no positive hostname scoping (`waf.go:241`); `applyDiff` deletes every `[felhom-geo]` rule not in THIS box's desired set (`geosync.go:320`) | READY (M) — **blocks shared-zone onboarding** | — | With two customers in one zone: they overwrite each other's Global rule forever; one customer's country policy applies zone-wide; per-app rules collide by name; and disabling the feature for one (or the hub's `RemoveGeoRules`) wipes them all. Interim mitigation, no code: keep geo-restriction OFF for every shared-zone customer. Fix = namespace descriptions by `customer_id` + add `http.host ends_with ""` to both expressions — a TWO-REPO change (controller + hub `RemoveGeoRules`). Same audit §5.1 | CC | | **R-255** | Security & access | P3 | **The check that would catch a fourth secret-in-the-body covers 4 of 27 pages, and the cheap gate that covers all 36 templates is blind to the shape that actually shipped.** Filed 2026-08-08 while closing R-254, **because a partial guard reported as complete is worse than no guard — it stops the next person looking.** **Two nets, both measured.** **(1) `scripts/secret_in_markup_gate.py`** reads all 36 templates and convicts any `{{ … }}` naming a secret unless allowlisted with a reason. It catches `{{.RetrievalPassword}}` and `{{.InitialCreds.Password}}`, **and it catches a launder through a local variable** because the assignment itself names the secret (`{{$v := .InitialCreds.Password}}` is convicted — verified). **It is blind to a secret arriving under a NEUTRAL PAGE-DATA KEY** — `data["Tagline"] = creds.Password` then `{{.AppInfo.Tagline}}` passes it cleanly, also verified. **That is exactly the shape of R-254 site two** (`value="{{$val}}"` inside an `{{if eq .Type "secret"}}` branch), so the gate **would not have caught one of the three instances it was written for.** **(2) The runtime body assertion** — render the page and grep the response for a sentinel — catches every shape, including that one (demonstrated on the same planted leak the gate missed). But it needs each page's data to be constructible in a test, and **only 4 of 27 page templates have that today**: `settings_security`, `app_info`, `deploy`, `backups_restore` — the four that were touched by R-249/R-252/R-253/R-254 and therefore got their own tests. **The other 23 pages have no runtime coverage at all.** **What closing this needs, so the cost is not re-estimated:** a per-page data fixture for the remaining 23 (most need a wired `Server` — `stackMgr`, `backupMgr`, agent seams), then one table-driven test that renders each with a sentinel substituted for every string in its data and asserts the sentinel is absent. **That is real scaffolding, which is why it was NOT built inside R-254's session** rather than half-built and declared done. | **READY** — owner Viktor | — | — | operator | diff --git a/documentation/runbooks/break-glass-sheet.md b/documentation/runbooks/break-glass-sheet.md index bc9f0633..a730c886 100644 --- a/documentation/runbooks/break-glass-sheet.md +++ b/documentation/runbooks/break-glass-sheet.md @@ -45,7 +45,7 @@ from the printed `data` field (hub-DB runbook Step 0 note) — `key show` cannot | S2 | Hub-DB off-site key — opens the hub database (ep0 `operator`, `host/dooplex-hub`) | PBS key `b2:19:bf:36:3b:97:3d:6c…` | `data`: ______________________________________________ | | S4 | Hub seal key `OFFSITE_SECRET_KEY` (64 hex) | — | ______________________________________________________ | | S5 | DooPlex restic / Secrets-export passphrase | — | ______________________________________________________ | -| S6a | Signing RECOVERY key `felhom-rec-recovery` | `SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k` | (staple the printout) | +| S6a | Signing RECOVERY key `felhom-rec-recovery` (also in the S1 copy since 2026-10-09; the paper is the copy that needs nothing else) | `SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k` | (staple the printout) | | S6b | Signing OPERATIONAL key `felhom-op-operational` | `SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw` | (staple the printout) | | S7 | ep0 read-only token `dooplex-hub@pbs!restore` | — | ______________________________________________________ | | S8 | Hetzner account: login e-mail · password · two-factor recovery codes | — | ______________________________________________________ | diff --git a/documentation/runbooks/total-loss-of-dooplex.md b/documentation/runbooks/total-loss-of-dooplex.md index 4caebb65..0673fc60 100644 --- a/documentation/runbooks/total-loss-of-dooplex.md +++ b/documentation/runbooks/total-loss-of-dooplex.md @@ -20,8 +20,7 @@ build tree. **Assume all of it is gone, and the house with it.** | The operator's workstation (if it has them) | the WireGuard peer to ep0 (`10.77.0.250`), maybe one of ep0's two root SSH keys, maybe a Bitwarden client's offline vault cache | the operator | **Lost with DooPlex and NOT in any copy** (measured 2026-10-09): the container registry (rebuild the images from -code); **the operator signing keys** `felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` -(`/mnt/5_hdd/felhom.eu/`, no passphrase, no backup — R-924); DooPlex's own SSH key (one of ep0's two root keys); +code); *(the operator signing keys were here until 2026-10-09 afternoon; they now ride the nightly copy — R-924)*; DooPlex's own SSH key (one of ep0's two root keys); the build tree and drills (`/mnt/5_hdd/felhom.eu/{build,drills}`); `.claude-memory` (same-disk restic only); the local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom; Longhorn on DooPlex only). @@ -38,7 +37,7 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom | **S3** | **Vaultwarden master password** (user count 1; **no two-factor rows**, SSO-linked to Authentik but password login allowed) | every item in the password manager | the operator's head | no — if remembered | | **S4** | **Hub seal key** `OFFSITE_SECRET_KEY` | the hub opening its console passwords and off-site passwords | DooPlex (`Secret/offsite-secret-key`); inside the Secrets export (behind S1 + S5); Vaultwarden | yes unless on paper or S1+S5 work | | **S5** | **DooPlex restic/GPG passphrase** (`/etc/backup/restic-password`) | opening the nightly Secrets export (`secrets/*.gpg`: every k8s Secret — Resend, report-api, Hetzner token, the hub's SSH keys, gitea-creds, TLS …) | DooPlex (`sdb1`); "an offline copy, out of band" (operator, recon 2026-08-06 — form unknown to CC). **Checked 2026-10-09:** the newest export opens with it (symmetric GPG, AES256) and holds 428 Secrets, among them all eight the hub uses (names only read) | depends on that offline copy | -| **S6** | **Signing recovery key** `felhom-rec-recovery` (and the operational key `felhom-op-operational`) | signing agent updates / config bundles / OS steps for every box; the recovery key authorizes rotating the operational key | **DooPlex only**, no passphrase, in no backup | **YES** — then every box needs on-site re-enrollment to accept a new key (`04` §4) | +| **S6** | **Signing recovery key** `felhom-rec-recovery` (and the operational key `felhom-op-operational`) | signing agent updates / config bundles / OS steps for every box; the recovery key authorizes rotating the operational key | DooPlex, no passphrase; **since 2026-10-09 (R-924 option A) also in the nightly copy `host/dooplex-gitea` (behind S1)** and, once printed, on the sheet | no, if S1 or the printout survives — otherwise every box needs on-site re-enrollment (`04` §4) | | **S7** | **ep0 read-only restore token** `dooplex-hub@pbs!restore` | reading both copies without ep0 root | DooPlex only (`/etc/felhom-hub-backup/token-restore`) | yes — but ep0 root can mint a new one (hub-DB runbook Step 2) | | **S8** | **Hetzner account login + its two-factor** | ep0's console (rescue, root reset), ep0 itself, Storage Boxes, the Hetzner API token | the operator; probably Vaultwarden | **circle**: if only in Vaultwarden, it is behind the copy it is needed to reach — paper or head | | S9 | ep0 root SSH | the PBS tunnel; minting tokens | ep0 `authorized_keys`: DooPlex's key (lost) + one more key, owner not recorded (operator workstation?) | partly — Hetzner console (S8) is the fallback | @@ -72,7 +71,7 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom (§3 step 1: the `MAIL-HOLD` marker, from the next hub release; until then: no network until the pending notices are understood). 9. **DNS:** in Cloudflare (S11) point `hub.felhom.eu` (today a CNAME to `dooplex.hopto.org`) and `gitea.dooplex.hu` at the new place. The boxes reconnect by themselves: their API keys are in the restored hub DB. -10. **Signing:** with S6 on paper, put the keys back and sign as before. Without it, no box accepts an agent update or a +10. **Signing:** the keys are in the restored copy (`signing/`) and on the sheet (S6) — put them back and sign as before. Without it, no box accepts an agent update or a bundle until it is re-enrolled on site (`04` §4). 11. **Backups again:** reinstall `scripts/hub-db-backup/` and `scripts/dooplex-offsite/` from the restored code, the tunnel and the tokens, and check the next night's copies. diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index c6a5a847..fa405d82 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,10 @@ +## 2026-10-09 (late afternoon) — dooplex-offsite carries the operator signing keys (R-924, operator ruling option A) + +- `felhom-dooplex-offsite`: `felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` and their `.pub` (from + `/mnt/5_hdd/felhom.eu/`) ride the nightly encrypted copy under `signing/`; a missing key or a link refuses the push. +- `felhom-dooplex-offsite-restore-test`: each private key must derive its own public half (`ssh-keygen -y`), never printed. +- 4 new tests (30 total), green with GNU and BusyBox tools; 3 red-proofs in `audits/dooplex-survival-2026-10-09/signing/`. + ## 2026-10-09 (afternoon) — dooplex-offsite carries the password manager too (R-923) - `felhom-dooplex-offsite`: a new step runs Vaultwarden's own `vaultwarden backup` (SQLite `VACUUM INTO`, consistent diff --git a/scripts/dooplex-offsite/felhom-dooplex-offsite b/scripts/dooplex-offsite/felhom-dooplex-offsite index a1d61386..60ba0c55 100755 --- a/scripts/dooplex-offsite/felhom-dooplex-offsite +++ b/scripts/dooplex-offsite/felhom-dooplex-offsite @@ -18,6 +18,8 @@ STATE=${FELHOM_DXOFF_STATE:-/var/lib/felhom-dooplex-offsite} TEXTFILE_DIR=${FELHOM_DXOFF_TEXTFILE_DIR:-/var/lib/node_exporter/textfile_collector} DUMPS=${FELHOM_DXOFF_DUMPS:-/mnt/5_hdd/backup/postgresql/dumps} SECRETS=${FELHOM_DXOFF_SECRETS:-/mnt/5_hdd/backup/secrets/exports} +SIGNING=${FELHOM_DXOFF_SIGNING:-/mnt/5_hdd/felhom.eu} +SIGNING_KEYS="felhom-op-operational felhom-rec-recovery felhom_op_ed25519" DUMP_MAX_AGE_H=${FELHOM_DXOFF_DUMP_MAX_AGE_H:-7} SECRETS_MAX_AGE_H=${FELHOM_DXOFF_SECRETS_MAX_AGE_H:-30} NOW=${FELHOM_DXOFF_NOW:-$(date +%s)} @@ -32,10 +34,11 @@ V() { kubectl -n vaultwarden-system exec deploy/vaultwarden -- "$@"; } umask 077 STAGE="$STATE/stage" mkdir -p "$STATE"; chmod 700 "$STATE" -rm -rf "$STAGE"; mkdir -p "$STAGE/root/gitea" "$STAGE/root/db" "$STAGE/root/secrets" "$STAGE/root/vaultwarden" +rm -rf "$STAGE"; mkdir -p "$STAGE/root/gitea" "$STAGE/root/db" "$STAGE/root/secrets" "$STAGE/root/vaultwarden" "$STAGE/root/signing" cleanup() { for f in "$STAGE/root/gitea/gitea/conf/app.ini" "$STAGE/root/db/gitea.dump" "$STAGE/root/db/globals.sql" \ - "$STAGE/root/vaultwarden/db.sqlite3" "$STAGE/root/vaultwarden/rsa_key.pem"; do + "$STAGE/root/vaultwarden/db.sqlite3" "$STAGE/root/vaultwarden/rsa_key.pem" \ + "$STAGE/root/signing/felhom-op-operational" "$STAGE/root/signing/felhom-rec-recovery" "$STAGE/root/signing/felhom_op_ed25519"; do [ -f "$f" ] && [ ! -L "$f" ] && { shred -u "$f" 2>/dev/null || rm -f "$f"; } done rm -rf "$STAGE" @@ -113,6 +116,17 @@ VUSERS=$(sqlite3 -readonly "$STAGE/root/vaultwarden/db.sqlite3" 'SELECT COUNT(*) echo "$VUSERS" > "$STAGE/root/vaultwarden/USERS" log "vaultwarden: integrity ok, $VUSERS user(s), files: db.sqlite3 $(echo $VLIST)" +# 3c. the operator signing keys (R-924, operator ruling 2026-10-09: option A — on paper AND in this copy). Without them +# no box accepts a signed agent update, config bundle or OS step after a loss of DooPlex. Each private key and its .pub; +# refuses when one is missing or is a link (a missing key would otherwise go unnoticed until the day it is needed). +for k in $SIGNING_KEYS; do + for f in "$k" "$k.pub"; do + [ -f "$SIGNING/$f" ] && [ ! -L "$SIGNING/$f" ] || die "signing key file $f missing (or a link) in $SIGNING" + cp "$SIGNING/$f" "$STAGE/root/signing/$f" + done +done +log "signing: $(echo $SIGNING_KEYS | wc -w | tr -d ' ') key(s) with their public halves" + # 4. the manifest the restore test checks, then the push echo "$GOT" > "$STAGE/root/REPOS" (cd "$STAGE/root" && find . -type f ! -name MANIFEST.sha256 -print0 | sort -z | xargs -0 sha256sum > MANIFEST.sha256) \ diff --git a/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test b/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test index a9fad36f..abe49420 100755 --- a/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test +++ b/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test @@ -21,7 +21,7 @@ die() { echo "felhom-dooplex-offsite-restore-test: FAILED: $*" >&2; exit 1; } umask 077 mkdir -p "$STATE"; chmod 700 "$STATE" T=$(mktemp -d "$STATE/restore.XXXXXX") -trap 'for f in "$T"/out/gitea/gitea/conf/app.ini "$T"/out/db/gitea.dump "$T"/out/db/globals.sql "$T"/out/vaultwarden/db.sqlite3 "$T"/out/vaultwarden/rsa_key.pem; do [ -f "$f" ] && shred -u "$f" 2>/dev/null; done; rm -rf "$T"' EXIT +trap 'for f in "$T"/out/gitea/gitea/conf/app.ini "$T"/out/db/gitea.dump "$T"/out/db/globals.sql "$T"/out/vaultwarden/db.sqlite3 "$T"/out/vaultwarden/rsa_key.pem "$T"/out/signing/felhom-op-operational "$T"/out/signing/felhom-rec-recovery "$T"/out/signing/felhom_op_ed25519; do [ -f "$f" ] && shred -u "$f" 2>/dev/null; done; rm -rf "$T"' EXIT export PBS_PASSWORD_FILE="$TOKENS/token-restore" PBS_FINGERPRINT LIST=$(proxmox-backup-client snapshot list host/dooplex-gitea --ns operator --output-format json --repository "$PBS_REPOSITORY_RESTORE") \ @@ -76,7 +76,15 @@ VUSERS=$(sqlite3 -readonly "$VDB" 'SELECT COUNT(*) FROM users;' 2>/dev/null) || VITEMS=$(sqlite3 -readonly "$VDB" 'SELECT COUNT(*) FROM ciphers;' 2>/dev/null) || die "cannot count Vaultwarden items" [ "${VUSERS:-0}" -gt 0 ] && [ "$VUSERS" = "$(cat "$O/vaultwarden/USERS" 2>/dev/null)" ] || die "Vaultwarden users: $VUSERS, USERS says $(cat "$O/vaultwarden/USERS" 2>/dev/null)" [ "${VITEMS:-0}" -gt 0 ] || die "the Vaultwarden copy holds no item" -log "checked: $FILES files match the manifest, $GOT repositories pass git fsck, gitea.dump readable, $(ls "$O"/secrets | wc -l | tr -d ' ') secrets file(s), Vaultwarden $VUSERS user(s) / $VITEMS item(s)" +# The signing keys (R-924): each private key must still derive its own public half — a usable key, never printed. +NSIGN=0 +for k in felhom-op-operational felhom-rec-recovery felhom_op_ed25519; do + [ -s "$O/signing/$k" ] && [ -s "$O/signing/$k.pub" ] || die "signing key $k missing from the copy" + DER=$(ssh-keygen -y -P '' -f "$O/signing/$k" 2>/dev/null | cut -d' ' -f1,2) || die "signing key $k does not open" + [ -n "$DER" ] && [ "$DER" = "$(cut -d' ' -f1,2 "$O/signing/$k.pub")" ] || die "signing key $k does not match its public half" + NSIGN=$((NSIGN + 1)) +done +log "checked: $FILES files match the manifest, $GOT repositories pass git fsck, gitea.dump readable, $(ls "$O"/secrets | wc -l | tr -d ' ') secrets file(s), Vaultwarden $VUSERS user(s) / $VITEMS item(s), $NSIGN signing key(s) match" TMP="$TEXTFILE_DIR/felhom_dooplex_offsite_restore.prom.$$" { diff --git a/scripts/dooplex-offsite/test_dooplex_offsite.py b/scripts/dooplex-offsite/test_dooplex_offsite.py index 9f397ae4..fc38ea20 100644 --- a/scripts/dooplex-offsite/test_dooplex_offsite.py +++ b/scripts/dooplex-offsite/test_dooplex_offsite.py @@ -106,6 +106,24 @@ try: except Exception as e: print("Error: " + str(e), file=sys.stderr); sys.exit(1) ''' +# ssh-keygen stand-in (the CI runner may lack OpenSSH): `ssh-keygen -y -P '' -f ` prints the public line a +# test key file carries after "PUB:" — the same contract as the real tool (derive the public half from the private). +FAKE_SSH_KEYGEN = r'''#!/bin/sh +[ "$1" = "-y" ] || exit 97 +f=""; while [ $# -gt 0 ]; do [ "$1" = "-f" ] && f=$2; shift; done +sed -n 's/^PUB://p' "$f" | grep . || { echo "Load key: invalid format" >&2; exit 255; } +''' +SIGNING_KEYS = ("felhom-op-operational", "felhom-rec-recovery", "felhom_op_ed25519") + + +def make_signing(d): + os.makedirs(d, exist_ok=True) + for i, k in enumerate(SIGNING_KEYS): + pub = "ssh-ed25519 AAAATEST%d %s" % (i, k) + open(os.path.join(d, k), "w").write("-----BEGIN TEST KEY-----\nPUB:%s\n" % pub) + open(os.path.join(d, k + ".pub"), "w").write(pub + "\n") + + REAL_SQLITE3 = None if os.environ.get("FORCE_FAKE_SQLITE3") else shutil.which("sqlite3") @@ -159,7 +177,10 @@ class Base(unittest.TestCase): os.makedirs(self.vw) make_vw_db(os.path.join(self.vw, "db.sqlite3")) open(os.path.join(self.vw, "rsa_key.pem"), "w").write("-----TEST KEY-----\n") - fakes = [("kubectl", FAKE_KUBECTL), ("proxmox-backup-client", FAKE_PBS), ("pg_restore", FAKE_PG_RESTORE)] + self.signing = j("signing") + make_signing(self.signing) + fakes = [("kubectl", FAKE_KUBECTL), ("proxmox-backup-client", FAKE_PBS), ("pg_restore", FAKE_PG_RESTORE), + ("ssh-keygen", FAKE_SSH_KEYGEN)] if not REAL_SQLITE3: fakes.append(("sqlite3", FAKE_SQLITE3)) for name, body in fakes: @@ -183,7 +204,7 @@ class Base(unittest.TestCase): def env(self, **kw): e = dict(os.environ, PATH=self.bin + ":" + os.environ["PATH"], FAKE_POD_DATA=self.pod, FAKE_PBS_DIR=self.pbs, - FAKE_STATE=self.t, FAKE_VW_DATA=self.vw, FELHOM_DXOFF_CONF=self.conf, FELHOM_DXOFF_TOKENS=self.tokens, + FAKE_STATE=self.t, FAKE_VW_DATA=self.vw, FELHOM_DXOFF_SIGNING=self.signing, FELHOM_DXOFF_CONF=self.conf, FELHOM_DXOFF_TOKENS=self.tokens, FELHOM_DXOFF_STATE=self.state, FELHOM_DXOFF_TEXTFILE_DIR=self.text, FELHOM_DXOFF_DUMPS=self.dumps, FELHOM_DXOFF_SECRETS=self.secrets) e.update({k: str(v) for k, v in kw.items()}) @@ -314,6 +335,21 @@ class Push(Base): self.assertIn("Vaultwarden integrity_check", r.stderr) self.assertEqual(self.pushed(), []); self.assertFalse(self.signal()) + def test_signing_keys_ride_the_copy(self): + r = self.push() + self.assertEqual(r.returncode, 0, r.stderr) + snap = os.path.join(self.pbs, "snaps", self.pushed()[0]) + for k in SIGNING_KEYS: + self.assertTrue(os.path.isfile(os.path.join(snap, "signing", k)), k) + self.assertTrue(os.path.isfile(os.path.join(snap, "signing", k + ".pub")), k) + + def test_a_missing_signing_key_refuses(self): + os.remove(os.path.join(self.signing, "felhom-rec-recovery")) + r = self.push() + self.assertNotEqual(r.returncode, 0) + self.assertIn("signing key file felhom-rec-recovery missing", r.stderr) + self.assertEqual(self.pushed(), []); self.assertFalse(self.signal()) + def test_failed_push_writes_no_signal(self): r = self.push(FAKE_PBS_FAIL=1) self.assertNotEqual(r.returncode, 0) @@ -378,6 +414,25 @@ class RestoreTest(Base): self.assertIn("Vaultwarden 1 user(s) / 3 item(s)", r.stdout) self.assertNotIn("encrypted", r.stdout + r.stderr) + def test_restore_checks_every_signing_key_opens_and_matches(self): + self.pushed_copy() + r = self.restore() + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn("3 signing key(s) match", r.stdout) + self.assertNotIn("BEGIN TEST KEY", r.stdout + r.stderr) + + def test_restore_refuses_a_signing_key_that_does_not_match(self): + snap = self.pushed_copy() + p = os.path.join(snap, "signing", "felhom-op-operational.pub") + open(p, "w").write("ssh-ed25519 AAAAOTHER someone-else\n") + man = os.path.join(snap, "MANIFEST.sha256") + kept = [l for l in open(man).readlines() if not l.rstrip().endswith("felhom-op-operational.pub")] + open(man, "w").writelines(kept) + r = self.restore() + self.assertNotEqual(r.returncode, 0) + self.assertIn("does not match its public half", r.stderr) + self.assertFalse(self.signal("felhom_dooplex_offsite_restore.prom")) + def test_restore_without_vaultwarden_fails(self): snap = self.pushed_copy() os.remove(os.path.join(snap, "vaultwarden/db.sqlite3"))