Day 2026-10-23 close-out: rulings D31-D36 in CONTEXT; STATUS (releases done, do list, D35 dated, D37/D38); register 191 -> 184 (closed R-49 R-982 R-985 R-991 R-1204 R-1205 R-1225 R-1226 R-1227 R-1230 R-1234; opened R-1235 R-1236 R-1237); the break-glass sheet current (D32, one A4, HU+EN); runbooks for D33/D34; poster facts; evidence day-2026-10-23
gates / gates (push) Successful in 12m53s
gates / gates (push) Successful in 12m53s
This commit is contained in:
admin committed
1 parent
52af38df6f
commit
2c9152d912
36 files changed
+565
-68
No files matched your search
@@ -93,6 +93,17 @@ itself. **Corrected:** one snapshot took 44 s on the live volume (0.63 s on a lo
|
||||
|
||||
### Step 4 — the push (on DooPlex, root; `felhom-hub-db-backup.service` + `.timer` 02:30) — DONE 2026-10-05
|
||||
|
||||
> **Since 2026-10-23 (D34, operator yes): every 6 hours.** The hub (≥ 0.167.0) writes a snapshot at 02:00, 08:00, 14:00
|
||||
> and 20:00 Budapest (`dbsnap.Slots`; start-up catch-up when the newest is older than 6 h); the timer pushes at 02:30,
|
||||
> 08:30, 14:30 and 20:30. The push waits (≤ 20 min) for a snapshot being written and refuses a newest snapshot older
|
||||
> than **7 h** (one slot gap + 1 h — a 2 h limit refused the hub's own catch-up once and mailed a false failure,
|
||||
> R-1234). Alarm `HubDBBackupStale` at **13 h** (two missed pushes). **Retention is unchanged and server-side on ep0**
|
||||
> (`prune-operator-hubdb`, 03:45, keep-daily 14, keep-weekly 8): the current day's copies (up to 4) live until the next
|
||||
> prune, then the newest of each day for 14 days and one per week for 8 weeks. One copy: ~500 MiB raw, ~25 MiB
|
||||
> uploaded. Proven 2026-10-23: a hand run, then the newest copy restored from ep0 with the read-only token was
|
||||
> byte-identical to the pod's snapshot, integrity ok, `SELECT COUNT(*) FROM hosts` = 4
|
||||
> (`audits/day-2026-10-23/13-hubdb-6h-D34.txt`). The text below is the original nightly design.
|
||||
|
||||
**The real script is `scripts/hub-db-backup/felhom-hub-db-backup`** (versioned, R-231; installed by `install.sh`;
|
||||
15 tests in `test_hub_db_backup.py`, run by hand — not in CI). It adds to the sketch below: it refuses a snapshot older
|
||||
than 26 h (the hub stopped snapshotting), a copy whose size differs from the pod's file, and a copy with no hosts; it
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Break-glass sheet — the keys that must exist outside DooPlex (R-923)
|
||||
# Break-glass sheet — the keys that must exist outside DooPlex (R-923, D32)
|
||||
|
||||
> **A template. It holds NO key value, and none may ever be written into this file, a commit, a log or a chat.**
|
||||
> Print this page, then write or stick each value onto the paper copy only. Keep the paper away from home (DooPlex is
|
||||
@@ -33,31 +33,43 @@ captured line on some systems; strip it when typing the value back.
|
||||
**Check a print later without exposing it:** for S1/S2 the only reliable check is one restore with a key file rebuilt
|
||||
from the printed `data` field (hub-DB runbook Step 0 note) — `key show` cannot check a rebuilt key.
|
||||
|
||||
## Public fingerprints — to check you hold the right key (not secret; not on the printed page)
|
||||
|
||||
| # | Fingerprint |
|
||||
|---|---|
|
||||
| S1 | PBS key `93:03:bf:d7:1f:4c:9e:fe…` (`proxmox-backup-client key show`) |
|
||||
| S2 | PBS key `b2:19:bf:36:3b:97:3d:6c…` |
|
||||
| S6a | `SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k` |
|
||||
| S6b | `SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw` |
|
||||
| S19 | recipient `age1aw83ej37x9sck89xwt856ry0553kf5sng3w544an4et54yjdx5qq79utrv` |
|
||||
|
||||
Every place below was checked to exist on 2026-10-23 (existence only, no value read). The release copy on ep0 (D33)
|
||||
and the 6-hourly hub-DB copy (D34) add NO key: the release copy is not encrypted (it holds no secret), and the hub-DB
|
||||
copy uses S2 as before.
|
||||
|
||||
---
|
||||
|
||||
## The sheet (print from here)
|
||||
## The sheet (print from here — one A4 page)
|
||||
|
||||
**FELHOM — break-glass keys.** Printed on: ____________ Stored at: ______________________________
|
||||
**FELHOM — vészhelyzeti kulcsok / break-glass keys.** Nyomtatva / printed: ____________ Hol tárolom / kept at: ______________________
|
||||
|
||||
| # | Key | Public fingerprint (to match the right key) | Value — write or stick here |
|
||||
Csak nevek és helyek; az értéket kézzel írd be. *Names and places only; write each value by hand.* Tartsd az otthonon kívül. *Keep it outside the home.*
|
||||
|
||||
| # | Név / Name | Hol van most / Where it lives now | Érték / Value |
|
||||
|---|---|---|---|
|
||||
| S1 | DooPlex off-site key — opens Gitea, the password manager, the k8s Secrets export (ep0 `operator`, `host/dooplex-gitea`) | PBS key `93:03:bf:d7:1f:4c:9e:fe…` | `data`: ______________________________________________ |
|
||||
| S2 | Hub-DB off-site key — opens the hub database (ep0 `operator`, `host/dooplex-hub`) | PBS key `b2:19:bf:36:3b:97:3d:6c…` | `data`: ______________________________________________ |
|
||||
| S4 | Hub seal key `OFFSITE_SECRET_KEY` (64 hex) | — | ______________________________________________________ |
|
||||
| S5 | DooPlex restic / Secrets-export passphrase | — | ______________________________________________________ |
|
||||
| S6a | Signing RECOVERY key `felhom-rec-recovery` (also in the S1 copy since 2026-10-09; the paper is the copy that needs nothing else) | `SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k` | (staple the printout) |
|
||||
| S6b | Signing OPERATIONAL key `felhom-op-operational` | `SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw` | (staple the printout) |
|
||||
| S7 | ep0 read-only token `dooplex-hub@pbs!restore` | — | ______________________________________________________ |
|
||||
| S8 | Hetzner account: login e-mail · password · two-factor recovery codes | — | ______________________________________________________ |
|
||||
| S11 | Cloudflare account: login · password · two-factor recovery codes | — | ______________________________________________________ |
|
||||
| S12 | Gmail `felhom.eu@gmail.com`: password · two-factor recovery codes | — | ______________________________________________________ |
|
||||
| S19 | CC secrets age key — opens `admin/cc-secrets` (every secret Claude Code uses on DooPlex); the store is in Gitea, behind S1 | recipient `age1aw83ej37x9sck89xwt856ry0553kf5sng3w544an4et54yjdx5qq79utrv` | `AGE-SECRET-KEY-1`______________________________________ |
|
||||
| S3 | Vaultwarden master password — **in your head**; write it here only if you decide to | — | ______________________________________________________ |
|
||||
| S1 | DooPlex off-site kulcs / DooPlex off-site key (Gitea, jelszókezelő / password manager, k8s Secrets) | DooPlex `/etc/felhom-dooplex-offsite/enc.key` | `data`: ____________________________ |
|
||||
| S2 | Hub-adatbázis off-site kulcs / Hub-DB off-site key | DooPlex `/etc/felhom-hub-backup/enc.key` | `data`: ____________________________ |
|
||||
| S4 | Hub pecsétkulcs / hub seal key `OFFSITE_SECRET_KEY` | k8s Secret `felhom-system/offsite-secret-key` | __________________________________ |
|
||||
| S5 | DooPlex restic / Secrets-export jelmondat / passphrase | DooPlex `/etc/backup/restic-password` | __________________________________ |
|
||||
| S6a | Aláíró HELYREÁLLÍTÓ kulcs / signing RECOVERY key | DooPlex `/mnt/5_hdd/felhom.eu/felhom-rec-recovery` | (tűzd ide / staple) |
|
||||
| S6b | Aláíró ÜZEMI kulcs / signing OPERATIONAL key | DooPlex `/mnt/5_hdd/felhom.eu/felhom-op-operational` | (tűzd ide / staple) |
|
||||
| S7 | ep0 csak-olvasó token / read-only token `dooplex-hub@pbs!restore` | DooPlex `/etc/felhom-hub-backup/token-restore` | __________________________________ |
|
||||
| S8 | Hetzner fiók / account (e-mail · jelszó / password · 2FA kódok / codes) | Vaultwarden | __________________________________ |
|
||||
| S11 | Cloudflare fiók / account (login · jelszó / password · 2FA kódok / codes) | Vaultwarden | __________________________________ |
|
||||
| S12 | Gmail `felhom.eu@gmail.com` (jelszó / password · 2FA kódok / codes) | Vaultwarden | __________________________________ |
|
||||
| S19 | CC secrets age kulcs / key (`admin/cc-secrets`) | DooPlex `~/.config/sops/age/keys.txt`; Vaultwarden | `AGE-SECRET-KEY-1`__________________ |
|
||||
| S3 | Vaultwarden mesterjelszó / master password | a fejedben / in your head | (csak ha akarod / only if you choose) |
|
||||
|
||||
**Not secret, needed with the keys:**
|
||||
- ep0: `167.233.158.164` (Hetzner, `felhom-hetzner`); PBS datastore `felhom-offsite`, namespace `operator`;
|
||||
its certificate fingerprint `c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd`.
|
||||
- Restore order: `total-loss-of-dooplex.md` (in the restored Gitea, repo `felhom.eu`, `documentation/runbooks/`).
|
||||
**Print that page too** — the runbook itself is inside the copy it explains how to open.
|
||||
|
||||
**Re-print when** a key above is rotated, and once a year.
|
||||
**Nem titok, de kell / not secret, needed:** ep0 = `167.233.158.164` (Hetzner, `felhom-hetzner`), PBS `felhom-offsite`, névtér / namespace `operator`; tanúsítvány / certificate
|
||||
`c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd`. Kiadások másolata / release copy: ep0 `/srv/felhom-release-copy/data`.
|
||||
Sorrend / order: `total-loss-of-dooplex.md` — **nyomtasd ki azt is / print it too.** Újranyomtatás / re-print: kulcscserekor és évente / when a key rotates, and yearly.
|
||||
@@ -19,8 +19,7 @@ build tree. **Assume all of it is gone, and the house with it.**
|
||||
| The boxes themselves | they keep running and backing up; they cannot report, update or mail until the hub is back | — |
|
||||
| The operator's workstation (if it has them) | the WireGuard peer to ep0 (`10.77.0.250`), maybe one of ep0's two root SSH keys, maybe a Bitwarden client's offline vault cache | the operator |
|
||||
|
||||
**Lost with DooPlex and NOT in any copy** (measured 2026-10-09): the container registry (rebuild the images from
|
||||
code); *(the operator signing keys were here until 2026-10-09 afternoon; they now ride the nightly copy — R-924)*; DooPlex's own SSH key (one of ep0's two root keys);
|
||||
**Lost with DooPlex and NOT in any copy** (measured 2026-10-09): *(the container registry was here until 2026-10-23: the released images and agent packages now ride a nightly copy to ep0 — D33, „Restore the release copy" below; the rest of the registry — older tags, rc builds, non-Felhom images — is still lost)*; *(the operator signing keys were here until 2026-10-09 afternoon; they now ride the nightly copy — R-924)*; DooPlex's own SSH key (one of ep0's two root keys);
|
||||
the build tree and drills (`/mnt/5_hdd/felhom.eu/{build,drills}`); `.claude-memory` (same-disk restic only); the
|
||||
local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom; Longhorn on DooPlex only).
|
||||
|
||||
@@ -66,16 +65,48 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom
|
||||
copy, use that. Every later login comes from here.
|
||||
6. **The Secrets export:** `gpg --batch --passphrase-file <S5 file> -d secrets-<stamp>.yaml.gpg > secrets.yaml` —
|
||||
every k8s Secret of the old cluster (shred it after use).
|
||||
7. **Gitea:** `gitea-restore.md` (on a normal network this time). Then rebuild the images from the code into the new
|
||||
registry: hub, controller, agent (`RUNBOOK-manual-build.md`).
|
||||
7. **Gitea:** `gitea-restore.md` (on a normal network this time). Then put the released images and agent packages
|
||||
back into its registry from ep0 — „Restore the release copy" below (same digests; no rebuild). Rebuild from code
|
||||
(`RUNBOOK-manual-build.md`) only what the copy does not hold.
|
||||
8. **The hub:** k3s, then `RUNBOOK-hub-db-offsite-backup.md` §3 with S2 and S4 — **start it with mail held**
|
||||
(§3 step 1: the `MAIL-HOLD` marker — shipped in hub 0.145.0, 2026-10-10).
|
||||
9. **DNS:** in Cloudflare (S11) point `hub.felhom.eu` (today a CNAME to `dooplex.hopto.org`) and `gitea.dooplex.hu` at
|
||||
the new place. The boxes reconnect by themselves: their API keys are in the restored hub DB.
|
||||
10. **Signing:** the keys are in the restored copy (`signing/`) and on the sheet (S6) — put them back and sign as before. Without it, no box accepts an agent update or a
|
||||
bundle until it is re-enrolled on site (`04` §4).
|
||||
11. **Backups again:** reinstall `scripts/hub-db-backup/` and `scripts/dooplex-offsite/` from the restored code, the
|
||||
tunnel and the tokens, and check the next night's copies.
|
||||
11. **Backups again:** reinstall `scripts/hub-db-backup/`, `scripts/dooplex-offsite/` and `scripts/release-copy/` from
|
||||
the restored code, the tunnel and the tokens (the release copy needs a new read-only registry token and a new SSH
|
||||
key in ep0's `felhom-relcopy` `authorized_keys`), and check the next night's copies.
|
||||
|
||||
## Restore the release copy (D33, since 2026-10-23)
|
||||
|
||||
**What it is.** Every night at 01:40 DooPlex copies the newest 3 release tags of `felhom-controller`, `felhom-hub` and
|
||||
`felhom-samba`, and the newest 5 agent versions (binary + config bundle), to ep0 `/srv/felhom-release-copy/data`
|
||||
(`scripts/release-copy/`). Images are in skopeo's `dir:` layout — the registry's own bytes — so a push back gives the
|
||||
**same digest** as the release. Nothing in it is secret; it is not encrypted. Alarm: `FelhomReleaseCopyStale`.
|
||||
|
||||
1. **Get it off ep0** (ep0 root — step 2): `rsync -a root@167.233.158.164:/srv/felhom-release-copy/data/ ./release-copy/`
|
||||
(or work on ep0 itself). Then `cd release-copy && sha256sum -c --quiet SHA256SUMS` — every line must pass.
|
||||
2. **Images back into the new registry**, one per tag (log in to the new Gitea first: `skopeo login <gitea host>`):
|
||||
```bash
|
||||
for d in images/*/*; do repo=${d#images/}; repo=${repo%%/*}; tag=${d##*/}
|
||||
skopeo copy --all --preserve-digests "dir:$d" "docker://<gitea host>/admin/$repo:$tag"
|
||||
echo "$repo:$tag $(skopeo inspect --raw "docker://<gitea host>/admin/$repo:$tag" | sha256sum | cut -c1-64) want $(cut -c8- "$d/DIGEST")"
|
||||
done
|
||||
```
|
||||
The two hashes on each line must be equal.
|
||||
3. **Agent packages back** (Gitea's generic registry, a token with package write):
|
||||
```bash
|
||||
for d in agent/*; do v=${d#agent/}; for f in felhom-agent felhom-config-bundle.json; do
|
||||
curl -fsS -u admin:"$(cat token-file)" -X PUT --upload-file "$d/$f" \
|
||||
"https://<gitea host>/api/packages/admin/generic/felhom-agent/$v/$f"; done; done
|
||||
```
|
||||
Check each download's sha256 against `agent/<v>/<file>.sha256`.
|
||||
4. **Keep the registry's name** (`gitea.dooplex.hu`, step 9's DNS) — the boxes pull `gitea.dooplex.hu/admin/…` by name.
|
||||
|
||||
**Proven 2026-10-23** (`audits/day-2026-10-23/08-release-copy-D33.txt`): the copy was pulled back from ep0, every
|
||||
sha256 matched, every image re-verified against its digest, and controller 0.340.0 from it loaded on a scratch box
|
||||
(image ID = its config digest; a container ran). **Not yet walked:** step 2 into a real new registry, and step 3.
|
||||
|
||||
## Walked on paper (2026-10-09): what still needs DooPlex if the operator has only the sheet, memory and a new machine
|
||||
|
||||
@@ -90,7 +121,7 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom
|
||||
| the hub's seal key | S4 (or S1+S5) | sheet, or via S5 |
|
||||
| the hub DB | S2 (or Vaultwarden) | sheet, or via S3 |
|
||||
| sign a box update | S6 | **sheet — today exists on DooPlex only** |
|
||||
| hub image | the registry (lost) | rebuild from restored code — needs a Go/Docker build machine, no secret |
|
||||
| hub, controller, samba images; agent packages | the release copy on ep0 (D33, nightly since 2026-10-23) | ep0 root (S8/S9) → „Restore the release copy" — no secret, no rebuild |
|
||||
| Gitea `main` → `felhom.eu` website, ISO host | Cloudflare (S11), the build tree (lost) | S11 on the sheet; the build tree is rebuilt from code |
|
||||
| Authentik (S17) | not in any copy | not needed for the business; DooPlex homelab apps that use it are out of scope |
|
||||
|
||||
|
||||
Reference in new issue
Block a user