admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

RECON: what DooPlex's own backup actually protects (R-232) — read-only, nothing changed
gates / gates (push) Successful in 9s
Details

Browse Source 
Five sets, 14/14 successful runs in 14 days, and a file restored byte-identical from the data repo
-- the first time a restore has ever been demonstrated here.

What the failure matrix exposes: notify_failure is a no-op (NOTIFY_ON_FAILURE=true, webhook
commented out) so a failed backup tells nobody; NOTHING leaves the box (Longhorn's target is
DooPlex itself over NFS, the only outbound-looking cron pulls inbound for another project); the
backup tree is a single writable path with no append-only; two sets are same-disk (.claude-memory
and the postgres dumps, whose source sits inside the backup tree); Longhorn retains 1 generation;
secrets/restic-repo has never held a snapshot because backup-secrets.sh never calls restic; and
/opt/backup/docs/BACKUP-RESTORE.md does not exist though the unit advertises it.

NOT a finding: the restic passphrase. On-box copy is on a different disk from the backups and the
operator holds an offline copy out of band, so disk loss is recoverable. The residual is only that
it is operator-held rather than system-held escrow.

A stale lock in the data repo is MINE, from the previous task's verification. Fenced from clearing
it; recorded with its impact and the one command that fixes it.
This commit is contained in:
authentik Default Admin
2026-08-06 12:16:38 +02:00
parent b1bf1f271c
commit a1a542b9a7
4 changed files with 269 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/RECON-dooplex-backup-2026-08-06.md
+214
View File
@@ -0,0 +1,214 @@
# RECON — what DooPlex's own backup actually protects
**2026-08-06, read-only.** Nothing was modified, stopped, pruned or deleted. One scratch restore into
`/tmp/recon-restore-20260806-121056`, deleted afterwards. No customer machine touched.
**State at start:** no backup running, no restic process. `dooplex-backup.timer` **enabled + active**,
next fire 2026-08-07 03:18:49. The service showing `disabled` is **expected** for a timer-driven unit
and is **not a finding** — recorded here so nobody chases it later. Last run 03:06:07 → 03:12:51,
`Result=success`, `ExecMainStatus=0`.
---
## 1. Set inventory
Five sets run from `backup-all.sh`. Two more things write into the same tree and are **not** part of
that service. Every "last success" below is from the journal or the repo, not from a config line.
| Set | Source paths | Source dev | Mechanism | Destination | Dest dev | Schedule | Last success | Snaps | Oldest → newest | Verified? | Off-site |
|---|---|---|---|---|---|---|---|---|---|---|---|
| **K3s State** | `/var/lib/rancher/k3s/server/{db,token,node-token,cred,tls}` · `/etc/rancher/k3s` | `sdb1` | restic | `backup/k3s/restic-repo` | **`sda1`** | nightly 03:00 | 2026-08-06 03:06:07 | 26 (13+13) | 03-31 → 08-06 | **yes** — `check --read-data-subset=5%` every run | no |
| **Kubernetes Secrets** | k8s API (all secrets + configmaps, per-namespace) | n/a | **GPG files — NO restic** | `backup/secrets/exports/` | **`sda1`** | nightly | 2026-08-06 03:06 | **0** | 07-06 → 08-06 (96 files, 134 MB, 30-day `find -mtime +30 -delete`) | no | no |
| **Git Manifests** | Gitea repos via API token | n/a | clone/mirror | `backup/homelab-manifests/` | **`sda1`** | nightly | 2026-08-06 03:06 | — | — | no | no |
| **User Data** | `/mnt/4_hdd/data` · `/mnt/5_hdd/felhom.eu/git/.claude-memory` | `sdg2` · **`sda1`** | restic | `backup/data/restic-repo` | **`sda1`** | nightly | 2026-08-06 10:51:17 (manual) / 03:0x nightly | 14 | 03-31 → 08-06 | **no** | no |
| **PostgreSQL Dumps** | `/mnt/5_hdd/backup/postgresql/dumps` | **`sda1`** | restic | `backup/postgresql/restic-repo` | **`sda1`** | nightly | 2026-08-06 03:09:52 | 11 | 06-25 → 08-06 | **no** | no |
| *(separate)* **Longhorn PVCs** | 77 volumes, replicas under `/var/lib/longhorn` | `sdb1` | Longhorn backupstore | `nfs://192.168.0.180:/mnt/5_hdd/backup/longhorn-pvc` | **`sda1`** | daily 04:00 + weekly, **retain=1** | 2026-08-06 04:10 (sync 10:08) | 77 vols / 88 GB | — | no | no |
| *(separate)* **jarrs.eu** | Hetzner → DooPlex | remote | rsync **inbound** | `backup/jarrs.eu/` | `sda1` | cron 04:00 | 2026-08-06 04:00 | 33 files / 19 MB | — | n/a | **inbound only** |
**Dormant destination directories** (nothing writes them any more): `demo-felhom` 25 GB, newest
**2026-07-10** (27 days); `sparkyfitness` 652 KB, newest **2026-06-24** (43 days);
`secrets/restic-repo` — initialised 2026-01-05, **zero snapshots ever**.
Tree total ≈ 167 GB. `/mnt/5_hdd` is 24% used, 6.6 TB free — **no space pressure**.
**Per-set reliability: 14 runs in 14 days, all five sets `✓ completed successfully` every run.** The
routine works; the questions below are about what it would survive, not whether it runs.
---
## 2. The same-disk answer, stated directly
**The destination for every set is `/dev/sda1`** (WDC WD103KFBX-68CCLN0, serial `WD-B1036H8J`).
| Set | Source device | Same disk as destination? |
|---|---|---|
| K3s State | `sdb1` (Apacer SSD, root) | **No** — cross-disk |
| Longhorn PVCs | `sdb1` (replicas under `/var/lib/longhorn`) | **No** — cross-disk |
| User Data → `/mnt/4_hdd/data` | `sdg2` (ST3000VX010 `Z6A07P2G`) | **No** — cross-disk |
| **User Data → `.claude-memory`** | **`sda1`** | **YES — same physical disk** |
| **PostgreSQL Dumps** | **`sda1`** (`/mnt/5_hdd/backup/postgresql/dumps`) | **YES — and the source is *inside the backup tree*** |
| Kubernetes Secrets | k8s API (no disk source) | destination `sda1`, single copy |
| Git Manifests | Gitea (which is a Longhorn PVC on `sdb1`) | No — cross-disk |
**The operator's suspicion was right, and it is narrower than feared.** Two sets are same-disk:
the memory store (added 2026-08-06) and the PostgreSQL dumps. The postgres case is the stranger of
the two — the dump directory lives *inside* `/mnt/5_hdd/backup/`, so restic is deduplicating a copy
of files that are already in the backup tree, onto the same disk. It protects against a bad dump, not
against losing `sda1`.
Everything else is genuinely cross-disk. **But cross-disk is not off-site — see §4.**
---
## 3. Failure matrix
**"Demonstrated" means someone has actually done it and checked the result.** Only one cell in this
column is a yes, and it was earned today.
| Failure | What is lost | What is recoverable | From where | How long | Ever demonstrated? |
|---|---|---|---|---|---|
| **A file deleted by mistake** | the file | **all of it**, to the previous night (or to the 6-hourly dump for a DB) | the relevant restic repo / Longhorn backup on `sda1` | minutes | **YES — today.** A file was restored from the `data` repo and matched the live original byte for byte (§5) |
| **`sda1` (5_hdd) fails** | **every backup, all of it** — all four restic repos, the Longhorn backupstore, the secrets exports, the manifest mirror | the *live system* keeps running: k3s, Longhorn replicas, Gitea, the registry and the hub are all on `sdb1`. `/mnt/4_hdd/data` survives on `sdg2`. **`.claude-memory` is lost** (source and only copy were both on `sda1`) | nothing — there is no second copy of the backups | n/a for the backups; the live system is unaffected | no |
| **`sdg2` (4_hdd) fails** | the live `/mnt/4_hdd/data` | **all of it** | `data` restic repo on `sda1` | hours (405 GiB) | **partially** — the single-file probe worked; a full-set restore has never been run |
| **DooPlex's board or PSU dies** | nothing permanently, if the disks survive | **everything**, by moving the disks or rebuilding and reattaching | `sda1` + `sdb1` in hand | days — and see the note below | no |
| **The flat burns** | **everything** — every disk is in the same box, and **nothing leaves it** | **nothing from this backup system.** Separately: customer data has a genuine off-site tier (Hetzner + ep0); the felhom *repos* exist on Gitea, which is on DooPlex; some repos have working clones on this same machine | — | — | no |
| **A bad script or ransomware writes over `/mnt/5_hdd/backup/`** | every backup at once | the live system only | nothing — the tree is a single writable path, and the restic repos are **not** `--append-only` | n/a | no |
**The note on "board or PSU dies":** recovery needs both the disks *and* the restic passphrase. The
on-box copy is `/etc/backup/restic-password` on **`sdb1`** — a different disk from the backups, which
is good. **The operator has confirmed an offline copy exists out of band**, so this row is
recoverable rather than fatal. See §6 for the one residual that remains.
---
## 4. Does anything leave the box?
**No. Nothing.** Stated without softening, because it is the row the matrix turns on.
- No `rclone`, no remote restic repository, no rsync-to-remote, no S3/B2/SFTP target anywhere in
`/opt/backup/scripts/`.
- The only outbound-looking job, `sync-hetzner-backups.sh` (cron 04:00), is **inbound**: it pulls
*jarrs.eu* database backups **from** Hetzner **to** DooPlex. It is a different project and it moves
data the wrong way to help here.
- Longhorn's backup target is `nfs://192.168.0.180:/mnt/5_hdd/backup/longhorn-pvc` — **DooPlex
writing to itself over NFS**, landing on `sda1`.
The felhom *product* has a real off-site chain (Hetzner Storage Box + ep0/PBS-DR) for customer boxes.
**DooPlex, which runs the hub that manages that chain, has none of it.**
---
## 5. The restore probe
| | |
|---|---|
| Repo | `/mnt/5_hdd/backup/data/restic-repo` |
| Snapshot | `b587f775` (2026-08-06 10:51:17) |
| File | `.claude-memory/hetzner-storagebox-api.md` (3.812 KiB) |
| Restored sha256 | `5b4d66b0341fe38c21b3a77f2f0af59ad4389fc21a8ce941c729fb807a3dd953` |
| Live sha256 | `5b4d66b0341fe38c21b3a77f2f0af59ad4389fc21a8ce941c729fb807a3dd953` |
| **Result** | **MATCH — a file came back, byte for byte** |
Scratch directory `/tmp/recon-restore-20260806-121056` **deleted**.
A first attempt used `MEMORY.md` and the hashes **differed** — correctly: the restored copy was
17,977 bytes, the file's exact size *at snapshot time*, and the live file had since been edited to
18,078. That is the snapshot behaving as a point-in-time copy, not a fault, and the probe was redone
against a file this session never touched.
---
## 6. The password question
**Measured on the box:** `/etc/backup/restic-password` — 45 bytes, `0600 root:root`, on **`/dev/sdb1`**
(the root SSD, *not* the disk holding the backups). It is **not a source path in any set**. The same
file is the passphrase for the GPG-encrypted secrets exports (`--passphrase-file "${RESTIC_PASSWORD_FILE}"`),
so one secret opens both the restic repos and the secrets archive.
**Off the box:** the operator confirms **an offline copy exists out of band.** A recon on the machine
cannot observe that, and without it this section would have concluded that an `sdb1` loss makes every
repo permanently unopenable — which would have been wrong. The failure matrix above is written on the
corrected basis.
**The residual, stated narrowly and without alarm:** the offline copy is *operator-held*, not
*system-held*. The felhom product solves the equivalent customer problem with hub-vaulted escrow and a
zero-knowledge recovery code; the operator case has no equivalent mechanism, so recovery time in the
"board dies" and "flat burns" rows depends on that copy being **current** and **findable by someone
other than the operator**. Worth confirming both; not worth engineering around.
---
## 7. Coverage gaps
| Item | Status | Evidence |
|---|---|---|
| **Gitea repositories** | **covered** (twice) | `gitea-data` is a 50 Gi Longhorn PVC → backupstore; plus `backup-manifests.sh` mirrors repos via the API to `backup/homelab-manifests/` |
| **Container registry** | **covered** | part of `gitea-data` (Gitea's built-in registry) |
| **Gitea database** | **covered** | CNPG → 6-hourly dumps → `postgresql` restic repo |
| **k3s state** | **covered, and the only verified set** | `db`, `token`, `node-token`, `cred`, `tls` + `/etc/rancher/k3s`, `check --read-data-subset=5%` each run |
| **Longhorn volumes** | **covered, thinly** | 77 volumes, but **`retain=1`** — one generation per volume. A corruption noticed after the next run has no earlier copy |
| **Hub database** | **covered** | `hub-data` (1 Gi) is a Longhorn PVC |
| **TLS material / secrets** | **partially** | exported and GPG-encrypted nightly, 30-day retention, on `sda1` only, **never entered into any restic repo** — `backup-secrets.sh` contains no `restic` call at all |
| **`/opt/backup` itself** | **NOT covered** | on `sdb1`, in no set. The scripts that perform every backup are backed up by nothing (R-231) |
| **`~/.claude/settings.json`** | **NOT covered** | on `sdb1`, in no set. Now reconstructible via `install_workspace.py` |
| **ISO build inputs** | **NOT covered** | `/mnt/5_hdd/felhom.eu/build`, on `sda1`, in no set |
| **Local git clones** | **NOT covered — and circular** | `/mnt/5_hdd/felhom.eu/git` on `sda1`, in no set. Recoverable from Gitea — **which runs on this machine.** If DooPlex is gone, both copies are gone together |
| **`.claude-memory`** | covered, **same-disk only** | §2 |
---
## 8. Observations — noticed, not acted on
1. **`/opt/backup/docs/` is EMPTY.** The systemd unit advertises
`Documentation=file:///opt/backup/docs/BACKUP-RESTORE.md` and that file does not exist. The
restore procedure you would reach for while the machine is on fire is absent, and the unit points
at it anyway.
2. **`notify_failure` is a no-op.** `NOTIFY_ON_FAILURE="true"` but `NOTIFY_WEBHOOK_URL` is commented
out, and the function requires it to be non-empty. **A failed backup notifies nobody.** The five
sets have succeeded 14/14, so this has never mattered yet — which is exactly when it is cheapest
to fix.
3. **`secrets/restic-repo` exists and has never held a snapshot.** A directory that looks like a
backup and is not one. `backup-secrets.sh` never calls restic; its only match for the word is a
comment about the password file.
4. **A stale lock is sitting in the `data` repo, and it is MINE.** `pid 3524524`, non-exclusive,
created 11:20:26, process gone — left by a `timeout`-wrapped `restic ls` during the previous
task's verification. **I am fenced from clearing it (§4 of the task) and have not.** Its impact is
concrete: `restic backup` needs only a non-exclusive lock and will succeed, but `restic check`
already refused (§ the check run below), and `apply_retention` → `forget --prune` needs an
**exclusive** lock. With `set -euo pipefail` and the `ERR` trap, tonight's User Data set would
therefore fail — and per observation 2, silently. One command clears it:
`sudo RESTIC_PASSWORD_FILE=/etc/backup/restic-password restic -r /mnt/5_hdd/backup/data/restic-repo unlock`
5. **Integrity checks:** `k3s` and `postgresql` pass `restic check` cleanly; `secrets` is empty so
passes trivially; **`data` could not be checked** because of observation 4. Only `k3s` is verified
on a schedule — the others have never been verified except by hand.
6. **Two dormant destination directories** hold 25 GB (`demo-felhom`, last written 2026-07-10) and
652 KB (`sparkyfitness`, 2026-06-24). Nothing writes them; nothing prunes them.
7. **The postgres dumps are produced by a k8s CronJob** (`database-system/postgresql-backup`, every
6 h) that the backup service does not control. `backup-postgresql.sh` refuses only if the
directory is *missing* — if the CronJob stopped, the set would keep succeeding over stale dumps.
Currently fresh: newest dump `20260806-100000` at 12:00 today.
8. **`restic check --read-data-subset` was not run** beyond what `backup-k3s.sh` does itself, per the
task's fence. No repository has ever had its data blocks read back in full.
---
## 9. Recommendations, ranked — recommend only, nothing implemented
| # | Recommendation | Cost | Failure it removes |
|---|---|---|---|
| 1 | **Clear the stale lock in the `data` repo** (command in §8.4) | one command | tonight's User Data set failing, silently |
| 2 | **Give `notify_failure` somewhere to go** — the project already has a working transactional mail path (Resend, used by CI and the hub) | ~10 lines | a backup failing and nobody learning it. This is the cheapest item here and the one that makes every other failure visible |
| 3 | **Get one copy off the box.** Even one set — k3s + secrets + postgres dumps is ~1.2 GB — to the Storage Box or ep0 | small; the credentials and the tooling already exist | the "flat burns" row, which is currently total loss |
| 4 | **Make `/mnt/5_hdd/backup` non-trivially destroyable** — restic `--append-only` via a REST/rclone-served repo, or a second target the writer cannot reach | moderate | the ransomware / bad-script row |
| 5 | **Move the postgres dump directory out of the backup tree**, and add `.claude-memory` to a cross-disk destination | small | the two same-disk cases in §2 |
| 6 | **Raise Longhorn `retain` above 1** | disk only, and there is 6.6 TB free | a corruption noticed one day late having no earlier copy |
| 7 | **Write `/opt/backup/docs/BACKUP-RESTORE.md`**, or drop the `Documentation=` line | an hour | reaching for a restore procedure that is not there |
| 8 | **Do one full-set restore drill** and record it | half a day | the entire "Ever demonstrated?" column being empty |
| 9 | **Put the secrets exports into their empty restic repo**, or delete the repo so it stops looking like a backup | small | a directory that reads as protection and is not |
---
## 10. The one-sentence answer
**The backup runs reliably, restores a file correctly, and would survive any single disk failing —
but every copy it makes lives in one box, and if that box is gone, so is all of it.**