Off-site lock live: Parts D/E/F evidence, ep0 copy runbook, 06/07 facts, register (R-820/R-821/R-342 closed, R-825 opened+closed, R-95/R-822 narrowed, R-823/R-824/R-826/R-827/R-828/R-830 opened; 327 -> 330); hub window-sweep test (test-only)
gates / gates (push) Successful in 41s
gates / gates (push) Successful in 41s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -0,0 +1,8 @@
|
||||
## hub v0.127.0 rollout 2026-10-03T14:59:40Z
|
||||
startup: off-site secrets sealed at rest (4 legacy plaintext row(s) sealed now)
|
||||
startup: Off-site key registrar enabled; daily key check at 07:10 Budapest
|
||||
raw DB after (prefix, length only):
|
||||
demo-felhom enc:v1: 83
|
||||
demo-hp enc:v1: 83
|
||||
tester-1 enc:v1: 83
|
||||
Tester-2 enc:v1: 83
|
||||
@@ -19,3 +19,9 @@
|
||||
## restored:
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 433.945s
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/offsiteapply (cached)
|
||||
FAIL
|
||||
|
||||
## RPC4: an unreadable count recorded as a measured zero (the v0.289.0 shape, live on demo-felhom)
|
||||
=== RUN TestRunOffbox_UnreadableCountIsNotZero
|
||||
offbox_window_test.go:266: an unreadable count was recorded as a measured zero: 0
|
||||
--- FAIL: TestRunOffbox_UnreadableCountIsNotZero (0.00s)
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
demo-felhom before migration (controller 0.288.0, sftp transport, u629488-sub1): 11 snapshots
|
||||
after two chain runs on 0.289.0/0.289.1 (pinned): 13 snapshots
|
||||
@@ -0,0 +1,17 @@
|
||||
transport=rclone-pinned target=u629488-sub1@u629488-sub1.your-storagebox.de:/home/felhom-repo
|
||||
## count before
|
||||
13 snapshots
|
||||
## restore: one file from the newest snapshot 5dd1f0b9
|
||||
restoring <Snapshot 5dd1f0b9 of [/mnt/sys_drive/felhom-data/backups/primary/opengist] at 2026-10-03 15:17:27.992813997 +0000 UTC by root@demo-felhom> to /tmp/rt
|
||||
restored files: 2
|
||||
sample: /mnt/sys_drive/felhom-data/backups/primary/opengist/data-stamps.json (398 bytes)
|
||||
## check (exclusive lock — the weekly integrity job's op)
|
||||
|
||||
no errors were found
|
||||
## delete attempt through the box's key: forget 6ea85413 (the OLDEST)
|
||||
unable to remove <snapshot/6ea854132e> from the repository
|
||||
[0:48] 0.00% 0 / 1 files deleted
|
||||
blob not removed, server response: 403 Forbidden (403)
|
||||
rc-line done
|
||||
## count after
|
||||
13 snapshots
|
||||
@@ -0,0 +1,4 @@
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.288.0
|
||||
offbox files: applied_marker known_hosts repo_password ssh_key
|
||||
target: u629488-sub3@u629488-sub3.your-storagebox.de port=23 repo=/home/felhom-repo transport=sftp settings.snapshot_count=
|
||||
91 snapshots
|
||||
@@ -0,0 +1,8 @@
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.289.1
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.289.1 Up About a minute (healthy)
|
||||
2026/10/03 15:19:24 offsiteapply.go:148: [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
|
||||
2026/10/03 15:19:34 offsiteapply.go:148: [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.288.0 (we are 0.289.1), no managed update running
|
||||
2026/10/03 15:19:37 offsiteapply.go:148: [INFO] [offsite-apply] the hub installed key SHA256:16GpHoTF0WHLbj15Pog0mpmDMit4/BLFLRfIqmHJQc8 append-only on u629488-sub3@u629488-sub3.your-storagebox.de (fresh=false)
|
||||
2026/10/03 15:19:37 offsiteapply.go:148: [INFO] [offsite-apply] offsite configured append-only for u629488-sub3@u629488-sub3.your-storagebox.de:/home/felhom-repo (key SHA256:16GpHoTF0WHLbj15Pog0mpmDMit4/BLFLRfIqmHJQc8)
|
||||
2026/10/03 17:19:36 [INFO] offsitekeys: installed box key SHA256:16GpHoTF0WHLbj15Pog0mpmDMit4/BLFLRfIqmHJQc8 for demo-hp pinned append-only (u629488-sub3@u629488-sub3.your-storagebox.de, dropped 5 unpinned line(s)) in 938ms
|
||||
2026/10/03 17:19:37 [INFO] offsitekeys: box confirmed key SHA256:16GpHoTF0WHLbj15Pog0mpmDMit4/BLFLRfIqmHJQc8 for demo-hp; 0 other line(s) removed
|
||||
@@ -0,0 +1,24 @@
|
||||
2026/10/03 15:22:53 night_chain.go:77: [INFO] [night-chain] offsite: started
|
||||
2026/10/03 15:22:53 offbox.go:991: [INFO] [offbox] backup run started (9 app(s) toggled)
|
||||
2026/10/03 15:24:49 offbox.go:1466: [WARN] [offbox] backed up bentopdf (/mnt/sys_drive/felhom-data/backups/primary/bentopdf, 0 mandatory path(s)) — but the recovery unit carried NO database dump and NO volume tar, so this snapshot holds none of the app's data; the next run with a dump leg will replace it
|
||||
2026/10/03 15:25:12 offbox_window.go:178: [INFO] [offbox] retention skipped (after-run): no clean-up window now (weekly windows are off) — nothing deleted (decision 68)
|
||||
2026/10/03 15:25:15 offbox.go:1234: [INFO] [offbox] backup OK: 9 app(s) backed up, 100 snapshot(s), 2m19s
|
||||
2026/10/03 15:25:15 night_chain.go:82: [INFO] [night-chain] offsite: done in 2m23s
|
||||
2026/10/03 15:25:15 night_chain.go:93: [INFO] [night-chain] finished in 4m14s
|
||||
transport=rclone-pinned target=u629488-sub3@u629488-sub3.your-storagebox.de:/home/felhom-repo
|
||||
## count before
|
||||
100 snapshots
|
||||
## restore: one file from the newest snapshot c43f2d0e
|
||||
restoring <Snapshot c43f2d0e of [/mnt/sys_drive/felhom-data/backups/primary/opengist] at 2026-10-03 15:25:07.97694046 +0000 UTC by root@demo-hp> to /tmp/rt
|
||||
restored files: 2
|
||||
sample: /mnt/sys_drive/felhom-data/backups/primary/opengist/manifest.json (1767 bytes)
|
||||
## check (exclusive lock — the weekly integrity job's op)
|
||||
|
||||
no errors were found
|
||||
## delete attempt through the box's key: forget 05c3346a (the OLDEST)
|
||||
unable to remove <snapshot/05c3346abc> from the repository
|
||||
[0:48] 0.00% 0 / 1 files deleted
|
||||
blob not removed, server response: 403 Forbidden (403)
|
||||
rc-line done
|
||||
## count after
|
||||
100 snapshots
|
||||
@@ -0,0 +1,40 @@
|
||||
## POST /offsite/key-audit (operator, Basic auth) 2026-10-03T15:26:40Z
|
||||
[
|
||||
{
|
||||
"customer": "Tester-2",
|
||||
"lines": 0,
|
||||
"pinned": 0,
|
||||
"findings": null
|
||||
},
|
||||
{
|
||||
"customer": "demo-felhom",
|
||||
"lines": 1,
|
||||
"pinned": 1,
|
||||
"findings": null
|
||||
},
|
||||
{
|
||||
"customer": "demo-hp",
|
||||
"lines": 1,
|
||||
"pinned": 1,
|
||||
"findings": null
|
||||
},
|
||||
{
|
||||
"customer": "tester-1",
|
||||
"lines": 3,
|
||||
"pinned": 0,
|
||||
"findings": [
|
||||
{
|
||||
"Fingerprint": "SHA256:3UoXpMIvo9gat9AL1380UK39UGAtO2T2CBBllo8n3Mg",
|
||||
"Kind": "unpinned"
|
||||
},
|
||||
{
|
||||
"Fingerprint": "SHA256:Jri1gf2AGHCTj8cJrFSpRj8+BxdY64OQ5Rnms/tbOZI",
|
||||
"Kind": "unpinned"
|
||||
},
|
||||
{
|
||||
"Fingerprint": "SHA256:gAhxqeDkxAPTOOeKQDHBDNe/AYpARkxI8h7Rrc8pLD8",
|
||||
"Kind": "unpinned"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,5 @@
|
||||
hub=https://hub.felhom.eu keylen=64
|
||||
consume-password HTTP 410
|
||||
body: gone: the hub no longer serves the storage password; register the box's public key at /api/v1/offsite/register-key/
|
||||
lines mentioning password field: 1
|
||||
2026/10/03 17:26:59 [WARN] offsite consume-password called by demo-hp — retired (decision 69); the box must register its public key (controller >= 0.289.0)
|
||||
@@ -0,0 +1,22 @@
|
||||
grant: {"ok":true}
|
||||
HTTP 202
|
||||
2026/10/03 15:22:24 controller_image_retention.go:184: [INFO] [stacks] controller image retention: deleted gitea.dooplex.hu/admin/felhom-controller:0.287.0 (244595237106, 409MB) — older than the previous controller and no container uses it (decision 56)
|
||||
2026/10/03 15:22:24 controller_image_retention.go:115: [INFO] [stacks] controller image retention: pass over 3 controller image(s) — running 0.289.1, previous "0.288.0" (by version order (no swap record names one present)), 1 candidate(s), 1 deleted, the rest kept
|
||||
2026/10/03 15:25:12 offbox_window.go:178: [INFO] [offbox] retention skipped (after-run): no clean-up window now (weekly windows are off) — nothing deleted (decision 68)
|
||||
2026/10/03 15:25:15 offbox.go:1234: [INFO] [offbox] backup OK: 9 app(s) backed up, 100 snapshot(s), 2m19s
|
||||
2026/10/03 15:22:24 controller_image_retention.go:184: [INFO] [stacks] controller image retention: deleted gitea.dooplex.hu/admin/felhom-controller:0.287.0 (244595237106, 409MB) — older than the previous controller and no container uses it (decision 56)
|
||||
2026/10/03 15:22:24 controller_image_retention.go:115: [INFO] [stacks] controller image retention: pass over 3 controller image(s) — running 0.289.1, previous "0.288.0" (by version order (no swap record names one present)), 1 candidate(s), 1 deleted, the rest kept
|
||||
2026/10/03 15:25:12 offbox_window.go:178: [INFO] [offbox] retention skipped (after-run): no clean-up window now (weekly windows are off) — nothing deleted (decision 68)
|
||||
2026/10/03 15:25:15 offbox.go:1234: [INFO] [offbox] backup OK: 9 app(s) backed up, 100 snapshot(s), 2m19s
|
||||
2026/10/03 15:25:15 night_chain.go:93: [INFO] [night-chain] finished in 4m14s
|
||||
2026/10/03 15:32:05 offbox_window.go:199: [ERROR] [offbox] clean-up window 1: the fake-snapshot guard REFUSED — nothing deleted: the policy would remove snapshot c6b5c67b from 2026-10-03T15:24:51Z — younger than 8 days, which honest retention never does (R-822)
|
||||
2026/10/03 15:32:10 offbox.go:1234: [INFO] [offbox] backup OK: 9 app(s) backed up, 109 snapshot(s), 2m22s
|
||||
2026/10/03 15:32:10 night_chain.go:93: [INFO] [night-chain] finished in 4m18s
|
||||
2026/10/03 17:32:03 [WARN] offsitekeys: clean-up window 1 OPENED for demo-hp (key SHA256:16GpHoTF0WHLbj15Pog0mpmDMit4/BLFLRfIqmHJQc8, 109 snapshot(s), max 43 removed, closes by 2026-10-03T15:52:03Z, one-shot=true)
|
||||
2026/10/03 17:32:06 [INFO] offsitekeys: clean-up window 1 CLOSED for demo-hp: outcome=guard-refused, 109 -> 109 (drop 0, allowed 43)
|
||||
2026/10/03 17:32:06 [INFO] Operator email sent for demo-hp/offsite_prune_guard_refused
|
||||
## key check after window 1 2026-10-03T15:32:32Z
|
||||
Tester-2 lines 0 pinned 0 findings 0
|
||||
demo-felhom lines 1 pinned 1 findings 0
|
||||
demo-hp lines 1 pinned 1 findings 0
|
||||
tester-1 lines 3 pinned 0 findings 3
|
||||
@@ -0,0 +1,21 @@
|
||||
## copy contents per namespace (filesystem listing, read-only)
|
||||
ns/demo-felhom/ct/9201/2026-09-22T04:12:20Z
|
||||
ns/demo-felhom/ct/9201/2026-09-29T04:16:43Z
|
||||
ns/demo-hp/ct/9201/2026-09-24T20:06:25Z
|
||||
ns/demo-hp/ct/9201/2026-10-01T20:15:29Z
|
||||
|
||||
## one snapshot's archives (no decryption — the copy is ciphertext; the catalog needs the customer's key)
|
||||
|
||||
4096 .
|
||||
4096 ..
|
||||
5576 catalog.pcat1.didx
|
||||
1220 client.log.blob
|
||||
690 index.json.blob
|
||||
401 pct.conf.blob
|
||||
179176 root.pxar.didx
|
||||
|
||||
## same view on ep0 (source)
|
||||
ns/demo-felhom/ct/9201/2026-09-22T04:12:20Z
|
||||
ns/demo-felhom/ct/9201/2026-09-29T04:16:43Z
|
||||
ns/demo-hp/ct/9201/2026-09-24T20:06:25Z
|
||||
ns/demo-hp/ct/9201/2026-10-01T20:15:29Z
|
||||
@@ -0,0 +1,29 @@
|
||||
## first pull 2026-10-03T19:13:17Z
|
||||
----
|
||||
Syncing datastore 'felhom-offsite', namespace 'demo-felhom' into datastore 'ep0-copy', namespace 'demo-felhom'
|
||||
Created namespace demo-felhom
|
||||
Found 1 groups to sync (out of 1 total)
|
||||
[ct/9201]: 2026-09-22T04:12:20Z: start sync
|
||||
[ct/9201]: 2026-09-22T04:12:20Z/pct.conf.blob: sync archive
|
||||
[ct/9201]: 2026-09-22T04:12:20Z/root.pxar.didx: sync archive
|
||||
[ct/9201]: 2026-09-22T04:12:20Z/root.pxar.didx: downloaded 2.057 GiB (62.788 MiB/s)
|
||||
[ct/9201]: 2026-09-22T04:12:20Z/catalog.pcat1.didx: sync archive
|
||||
[ct/9201]: 2026-09-22T04:12:20Z/catalog.pcat1.didx: downloaded 652.916 KiB (10.225 MiB/s)
|
||||
[ct/9201]: Snapshot ct/9201/2026-09-22T04:12:20Z: got backup log file client.log.blob
|
||||
[ct/9201]: 2026-09-22T04:12:20Z: sync done
|
||||
[ct/9201]: percentage done: 50.00% (1/2 snapshots)
|
||||
[ct/9201]: 2026-09-29T04:16:43Z: start sync
|
||||
[ct/9201]: 2026-09-29T04:16:43Z/pct.conf.blob: sync archive
|
||||
[ct/9201]: 2026-09-29T04:16:43Z/root.pxar.didx: sync archive
|
||||
[ct/9201]: 2026-09-29T04:16:43Z/root.pxar.didx: downloaded 641.072 MiB (61.537 MiB/s)
|
||||
[ct/9201]: 2026-09-29T04:16:43Z/catalog.pcat1.didx: sync archive
|
||||
[ct/9201]: 2026-09-29T04:16:43Z/catalog.pcat1.didx: downloaded 768.894 KiB (16.864 MiB/s)
|
||||
[ct/9201]: Snapshot ct/9201/2026-09-29T04:16:43Z: got backup log file client.log.blob
|
||||
[ct/9201]: 2026-09-29T04:16:43Z: sync done
|
||||
[ct/9201]: percentage done: 100.00% (2/2 snapshots)
|
||||
Finished syncing namespace demo-felhom, current progress: 2 groups, 0 snapshots
|
||||
pull datastore 'ep0-copy' end
|
||||
TASK OK
|
||||
elapsed 201 s
|
||||
## bytes on DooPlex
|
||||
12G /mnt/5_hdd/backup/ep0-copy
|
||||
@@ -0,0 +1,10 @@
|
||||
+====================+================+==========+========+================+==========+==============+=========+=================================================================================+
|
||||
| id | sync-direction | store | remote | remote-store | schedule | group-filter | rate-in | comment |
|
||||
+====================+================+==========+========+================+==========+==============+=========+=================================================================================+
|
||||
| ep0-felhom-offsite | | ep0-copy | ep0 | felhom-offsite | 05:00 | all | | decision 70: nightly copy of ep0 felhom-offsite; never removes what ep0 removed |
|
||||
+====================+================+==========+========+================+==========+==============+=========+=================================================================================+
|
||||
+=================+==========+===========+=================+================+============================================+
|
||||
| id | store | schedule | ignore-verified | outdated-after | comment |
|
||||
+=================+==========+===========+=================+================+============================================+
|
||||
| verify-ep0-copy | ep0-copy | sat 06:30 | 1 | 30 | decision 70: weekly verify of the ep0 copy |
|
||||
+=================+==========+===========+=================+================+============================================+
|
||||
@@ -0,0 +1,37 @@
|
||||
## tunnel unit
|
||||
[Unit]
|
||||
Description=Felhom: SSH tunnel DooPlex 127.0.0.1:18007 -> ep0 PBS 127.0.0.1:8007 (decision 70, nightly pull-sync of felhom-offsite)
|
||||
Documentation=file:///mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/runbooks/ep0-datastore-copy.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
User=kisfenyo
|
||||
ExecStart=/usr/bin/ssh -N -o BatchMode=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -L 127.0.0.1:18007:127.0.0.1:8007 root@167.233.158.164
|
||||
Restart=always
|
||||
RestartSec=30
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
active
|
||||
|
||||
## notification target + matcher (password lives in notifications-priv.cfg, root:root 0600, not shown)
|
||||
smtp: felhom-operator
|
||||
author Felhom DooPlex PBS
|
||||
comment decision 70: ep0 copy job failures to the operator (Resend)
|
||||
from-address monitoring@felhom.eu
|
||||
mailto admin@felhom.eu
|
||||
mode tls
|
||||
port 465
|
||||
server smtp.resend.com
|
||||
username resend
|
||||
|
||||
matcher: felhom-operator-errors
|
||||
comment decision 70: any error (the ep0 pull-sync and verify jobs) reaches the operator
|
||||
match-severity error
|
||||
mode all
|
||||
target felhom-operator
|
||||
|
||||
## test mail: received in the operator mailbox 2026-10-03T19:17:41Z, subject 'Test notification', from monitoring@felhom.eu to admin@felhom.eu (Gmail connector search)
|
||||
|
||||
## ep0 side: token root@pam!dooplex-sync, ACL DatastoreReader on /datastore/felhom-offsite (propagate) — the only change on ep0
|
||||
Reference in New Issue
Block a user