felhom.eu/REPORT-hub-db-offsite-2026-10-05.md

REPORT — the hub database off DooPlex (R-173 option A) and the torn-backup live test (R-519) — 2026-10-05 (evening)

Part	Result	Releases / changes
A — PVC label + nightly VACUUM INTO snapshot	done — label enabled, volume 2 Gi, snapshot 353 MiB in 44 s, keep 2, mutex; 6 red-proofs	hub v0.136.0 (one release)
B — ep0: namespace, two tokens, prune job	done — operator ns, dooplex-hub@pbs with !push (DatastoreBackup) and !restore (DatastoreReader), prune-operator-hubdb (ns operator only); household jobs unchanged	ep0 config only
C — DooPlex push / restore-test units, key, alarm	done — versioned scripts + units, 15 tests, 9 red-proofs; enc.key made, operator saved the paper key; two alarm rules, promtool-proven, 2 red-proofs	scripts/hub-db-backup/, homelab-manifests rules
D — live proof	done — push, ep0 listing, restore test, token limits, paper-key restore, alarm fired + mailed + cleared; R-519 live on 9202; runbook §3 tested (steps 1–3)	hub/cmd/hubdb-check (tool, not in the image)

Side events (all with the operator's word): the Longhorn instance-manager on DooPlex was restarted (77/77 volumes healthy in 110 s); my failed offline-grow attempt kept the hub down ~9.5 min (12:53–13:02Z); zipline came back on a newer release and was pinned to 4.7.0.

Wrong claims (in the brief and the runbook)

1. Runbook §1: "the PVC label is the source of truth (Longhorn syncs PVC → Volume), so the backups can stop at any sync." The Volume kept enabled through every sync since February while the PVC said disabled — Longhorn did not copy it down (partA/step1-labels.txt, before-state). Which one Longhorn reads was not measured; both say enabled now.
2. Brief + runbook: "keep 2 snapshots" on the 1 Gi volume. Two 353 MiB snapshots plus the 370 MB live database did not fit (594 MB free). The operator chose to grow to 2 Gi.
3. Runbook Step 3 implied the snapshot is quick (0.63 s measured on a scratch copy). On the live Longhorn volume: 44 s.
4. Runbook Step 2: --schedule 'daily 03:45' — not a PBS calendar event (unable to parse … 'daily'); 03:45 is.
5. Runbook Step 2: proxmox-backup-client namespace create … root@pam needs a password CC does not have; proxmox-backup-debug api create …/namespace works (the CLI then crashes printing the result — the namespace exists). user generate-token … --output-format json is refused; the API path is …/access/users/<u>/token/<name>.
6. Brief: the push token "DatastoreBackup only (no prune, no read)". No prune: TRUE (missing Datastore.Modify| Datastore.Prune). No read: FALSE — the push token restored its own copy (restore complete … 352.676 MiB): PBS lets a backup's owner read it. Scope: ns operator only, and the content is encrypted with a key ep0 never sees.
7. Runbook Step 2: a token's ACL alone grants it — PBS cuts a token's rights down by its user's, so the user needs both roles on the path too (done; the user has no password).
8. The 1 Gi → 2 Gi growth was assumed routine (allowVolumeExpansion=true). It failed: the instance-manager called a vanished host PID (nsenter: cannot open /host/proc/196610/ns/mnt), and offline growth was blocked by the expansion's own attachment ticket (partA/step1-expansion-failure.txt, step1-offline-expansion.txt).
9. Runbook §3 as proposed: "a working reveal proves the key matches" — it needs a running hub on the copy. A check that needs no hub now exists (hubdb-check), and the procedure says how to rebuild the key from the paper data field.
10. My own: "the hub is down ~2 minutes" for the offline grow — it was ~9.5 minutes, and it did not work.

Part A — evidence (documentation/audits/hub-db-offsite-2026-10-05/partA/)

• Architecture read first: 05-hub-architecture.md §16 (new §16.3), 07-backup-architecture.md, the runbook.
• Hub baseline cea8502f-era main at v0.135.0; commits d4be9f6 (code, docs, manifest), 5d8922e (image tag 0.136.0).
• internal/store/snapshot.go (SnapshotInto = VACUUM INTO ?), internal/dbsnap (.tmp → rename, 0600, keep 2, ErrBusy, NeedsCatchUp), cmd/hub/main.go (02:00 Budapest + start-up catch-up).
• Tests: dbsnap_test.go — integrity_check ok and equal row counts in EVERY table vs. the live DB with rows still in the WAL (precondition asserted: a plain file copy held 0 of 150 hosts); keep 2; never two at once; a failed write leaves no file; catch-up age. cmd/hub/r173_wiring_test.go (AST). Red-proofs R1–R6 convict (red-proof.txt, red-proof-run1.txt; R2's first run convicted by HANGING — the test now fails in 2 s).
• Live: db snapshot written: hub-20261005T123037Z.db (369807360 bytes, 44.276s), mode -rw-------.
• PVC/Volume labels both enabled; PVC capacity 2Gi; /data 2,028,392 KB, 37 % used (step1-im-restart.txt).
• Full hub suite green (go build/vet/test ./...).

Part B — evidence (partB/)

• Before (ep0-before.txt): prune jobs prune-demo-hp, prune-demo-felhom (ns each, keep-last 2, 03:30); GC sun 04:30; users root@pam, felhom@pbs; no operator ns.
• After (ep0-after.txt): the same two jobs unchanged + prune-operator-hubdb (ns operator, max-depth 0, 03:45, keep-daily 14, keep-weekly 8); GC unchanged; user dooplex-hub@pbs; tokens !push, !restore; ACLs only on /datastore/felhom-offsite/operator.
• Token secrets: ep0 → pipe → root 0600 files on DooPlex (36 bytes each), never on a command line or printed. No household namespace was listed or read (the operator check read only whether that one name exists).

Part C — evidence (partC/)

• Tools present: sqlite3 3.46.1, proxmox-backup-client 4.2.3, kubectl (k3s) as root, textfile dir /var/lib/node_exporter/textfile_collector (already scraped), tunnel felhom-ep0-pbs-tunnel active on 127.0.0.1:18007.
• scripts/hub-db-backup/ (commit cea8502f), installed with install.sh; config /etc/felhom-hub-backup/{env,token-push,token-restore,enc.key} all root 0600. Timers enabled after the manual runs: next push Tue 02:31, next restore test Sun 2026-10-11 04:31.
• enc.key created (kdf none, fingerprint b2:19:bf:36:…); STOPPED for the paper key; the operator saved the data field before the first push.
• Tests: 15 (test_hub_db_backup.py, fakes for kubectl and proxmox-backup-client). Red-proofs P1–P9 all convict (red-proof.txt; P4 first did NOT convict — masked by integrity_check — and P5 errored; both tests were strengthened). Not in CI (R-885).
• Alarm: homelab-manifests ebc14b0; promtool check rules (4 rules) and promtool test rules green; red: threshold 260 h → FAILED; absent() removed → FAILED (the first attempt at that mutation did not apply — recorded) (alarm-rule-test.txt, bf_test.yml). Only ConfigMap/prometheus-rules was synced; POST /-/reload → 200; rules listed in /api/v1/rules.

Part D — evidence (partD/)

• Push (push-1.txt): unit Result=success; checked: 369807360 bytes, integrity ok, 4 host(s); Encryption key fingerprint: b2:19:bf:36:…; 352.676 MiB (17.242 MiB compressed) in 7.19 s; success timestamp written.
• ep0 listing with the read-only token: host/dooplex-hub/2026-10-05T13:50:18Z 352.677 MiB.
• Restore test (restore-test-1.txt): checked: integrity ok, 4 host(s), 4 sealed console password(s), 0 readable; success timestamp written; no restored file left.
• Token limits + paper key (token-limits-and-paperkey.txt): forget refused for both tokens; restore token cannot back up; neither can list the datastore root; push token CAN restore its own copy (claim 6); a key file rebuilt from the data field only → restore rc=0, integrity ok, 4 hosts; without a key → missing key.
• Alarm (alarm-drill.txt): success file hidden 13:51:46Z → pending 13:52:06Z → firing 14:22Z → Alertmanager: 1 active alert, receiver email-notifications, notifications_total{email} 5 and failed_total 0 → a real push 14:23 (2 s) → inactive 14:23:42Z, email counter 6. Both mails arrived (operator's inbox screenshot, 2026-10-05): „[FIRING] HubDBBackupStale" 16:22 and „[RESOLVED] HubDBBackupStale" 16:27 CEST.
• R-519 on 9202 (r519/): 9202 raised to controller 0.296.0 by hand (00-upgrade-9202.txt); bookstack installed; run 1 complete (33 s); run 2 cut by docker restart felhom-controller at 14:05:59.67Z, 0.8 s after Volume dump: bookstack/bookstack_bookstack_config and before its database volume. After: the run record turned interrupted; the controller restarted bookstack itself; /backups and /backups/apps both carry data-interrupted-run ("A legutóbbi mentés (2026-10-05 16:05) megszakadt …"; negative control 0); the restore point reads 14:04:44Z — the run-1 database volume, its oldest part (config 14:05:58, SQL 14:05:54). Run 3 complete → notice gone from both pages, record clear, the torn .tar.tmp replaced. Bookstack removed through the product: 0 containers, 0 volumes, no folder, no backups.
• Runbook §3 (restore-procedure/drill.txt): restore from ep0 → hubdb-check with the seal key from the Secret (file → file) → hosts=4 console_passwords_opened=4 failed=0; a random key → opened=0 failed=4 FAILED. Steps 4–5 (into a live PVC) not run. hubdb-check red-proof R7 convicts.

Rows and register

• Closed: R-519 (live-proven) → CLOSED-ITEMS.md in the same commit.
• Narrowed: R-173 (option A in force; left: runbook §3 steps 4–5), R-232 ((b) and (a) partly, for the hub DB).
• R-231 not touched (it is about /opt/backup/scripts/); the new units are versioned from day one.
• Opened: R-882 (Longhorn stale host PID), R-883 (8 workloads on a moving tag; zipline pinned), R-884 (monitoring Prometheus Deployment OutOfSync), R-885 (scripts' Python tests not in CI), R-886 (Alertmanager cannot write nflog/silences since the restart).
• Register: 332 → 336. STATUS.md updated (Tonight section; the two old "needs you" items marked decided/done).

Teardown — three layers

• Machines: 9202 — bookstack removed through the product; its controller stays 0.296.0 (was 0.295.0); helper files in the guest removed. DooPlex — scratch restores shredded; the hubdb-check binary shredded; the units, timers, config and keys stay (they are the deliverable).
• Hosts: ep0 — the new user, tokens, namespace, prune job stay (the deliverable); nothing else changed. demo-hp — none.
• Hub: none provisioned. The hub runs v0.136.0.
• Secrets: a scan of 7,132 committed/working files against the 8 real secret values of this session → 0 hits. Scratch copies (demo password, bookstack deploy values, the zipline pre-change dump, rule files, used signed envelopes) shredded.

Unproven / say-so

• unproven.py --summary: walked 20, partial 17, built 14, missing 4 — NOT WALKED 35 of 55; no number moved (the hub-DB capability is a new row in 00 §G, not one of the 55 walked claims).
• Runbook §3 steps 4–5.