Off-site safety finished: Parts A-F evidence, decision 74, golden 0.290.0 recorded (vouched, floor 0.290.0), restore walked from the DooPlex copy, register 330 -> 328 (R-823/824/826/827/828/830 closed; R-833, R-834 opened)
gates / gates (push) Successful in 30s
gates / gates (push) Successful in 30s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
## Part A live window on demo-hp 2026-10-04T05:31:42Z
|
||||
grant: {"ok":true}
|
||||
BEFORE
|
||||
count: 118 snapshots
|
||||
plan (remove times, by minute):
|
||||
6 2026-10-03T15:24
|
||||
3 2026-10-03T15:25
|
||||
HTTP 202
|
||||
2026/10/04 05:33:13 controller_image_retention.go:184: [INFO] [stacks] controller image retention: deleted gitea.dooplex.hu/admin/felhom-controller:0.288.0 (3e9d251e1ab7, 410MB) — older than the previous controller and no container uses it (decision 56)
|
||||
2026/10/04 05:33:13 controller_image_retention.go:115: [INFO] [stacks] controller image retention: pass over 3 controller image(s) — running 0.290.0, previous "0.289.1" (by version order (no swap record names one present)), 1 candidate(s), 1 deleted, the rest kept
|
||||
2026/10/04 05:36:16 offbox_window.go:232: [INFO] [offbox] clean-up window 2: the policy removes nothing
|
||||
2026/10/04 05:36:22 offbox.go:1234: [INFO] [offbox] backup OK: 9 app(s) backed up, 127 snapshot(s), 2m24s
|
||||
2026/10/04 05:36:22 night_chain.go:93: [INFO] [night-chain] finished in 4m21s
|
||||
AFTER
|
||||
count: 127 snapshots
|
||||
plan (remove times, by minute):
|
||||
7 2026-10-04T02:16
|
||||
2 2026-10-04T02:17
|
||||
2026/10/04 07:36:14 [WARN] offsitekeys: clean-up window 2 OPENED for demo-hp (key SHA256:16GpHoTF0WHLbj15Pog0mpmDMit4/BLFLRfIqmHJQc8, 127 snapshot(s), max 63 removed, closes by 2026-10-04T05:56:14Z, one-shot=true)
|
||||
2026/10/04 07:36:17 [INFO] offsitekeys: clean-up window 2 CLOSED for demo-hp: outcome=nothing, 127 -> 127 (drop 0, allowed 63)
|
||||
## key check after window 2:
|
||||
Tester-2 lines 0 pinned 0 findings 0
|
||||
demo-felhom lines 1 pinned 1 findings 0
|
||||
demo-hp lines 1 pinned 1 findings 0
|
||||
tester-1 lines 0 pinned 0 findings 0
|
||||
## hub alarms for the window (drop/guard/failed) in the last 15 min: 0
|
||||
## weekly windows ON (fleet switch): {"ok":true}
|
||||
@@ -0,0 +1,22 @@
|
||||
## Part B 2026-10-04T05:14:08Z
|
||||
Prune job created: prune-ep0-copy
|
||||
created
|
||||
+================+=========+==========+====+==========+===========+===========+=============+============+=============+==============+=============+
|
||||
| id | disable | store | ns | schedule | max-depth | keep-last | keep-hourly | keep-daily | keep-weekly | keep-monthly | keep-yearly |
|
||||
+================+=========+==========+====+==========+===========+===========+=============+============+=============+==============+=============+
|
||||
| prune-ep0-copy | | ep0-copy | | 07:30 | | | | | 8 | | |
|
||||
+================+=========+==========+====+==========+===========+===========+=============+============+=============+==============+=============+
|
||||
| id | sync-direction | store | remote | remote-store | schedule | group-filter | rate-in | comment |
|
||||
| ep0-felhom-offsite | | ep0-copy | ep0 | felhom-offsite | 05:00 | all | | decision 70: nightly copy of ep0 felhom-offsite; never removes what ep0 removed |
|
||||
gc-schedule: sun 08:30
|
||||
|
||||
## the copy NOW (what a keep-weekly 8 prune would judge)
|
||||
ns/demo-felhom/ct/9201/2026-09-22T04:12:20Z
|
||||
ns/demo-felhom/ct/9201/2026-09-29T04:16:43Z
|
||||
ns/demo-hp/ct/9201/2026-09-24T20:06:25Z
|
||||
ns/demo-hp/ct/9201/2026-10-01T20:15:29Z
|
||||
dry-run, by reasoning (PBS has no CLI dry-run for a prune JOB): every group holds 2 snapshots in 2 different ISO weeks; keep-weekly 8 keeps the newest of each of the last 8 weeks that have one → both kept → NOTHING would be removed.
|
||||
|
||||
## space
|
||||
12G /mnt/5_hdd/backup/ep0-copy
|
||||
/dev/sda1 9.1T 3.2T 5.5T 37% /mnt/5_hdd
|
||||
@@ -0,0 +1,16 @@
|
||||
## Part C 2026-10-04T05:07:52Z
|
||||
before:
|
||||
Tester-2 lines 0 pinned 0 findings 0
|
||||
demo-felhom lines 1 pinned 1 findings 0
|
||||
demo-hp lines 1 pinned 1 findings 0
|
||||
tester-1 lines 3 pinned 0 findings 3
|
||||
remove-unpinned tester-1: {"changed":3}
|
||||
after (the daily check, by hand):
|
||||
Tester-2 lines 0 pinned 0 findings 0
|
||||
demo-felhom lines 1 pinned 1 findings 0
|
||||
demo-hp lines 1 pinned 1 findings 0
|
||||
tester-1 lines 0 pinned 0 findings 0
|
||||
2026/10/04 07:07:54 [ERROR] offsitekeys: audit tester-1: 3 line(s) can delete off-site history: unpinned SHA256:3UoXpMIvo9gat9AL1380UK39UGAtO2T2CBBllo8n3Mg; unpinned SHA256:Jri1gf2AGHCTj8cJrFSpRj8+BxdY64OQ5Rnms/tbOZI; unpinned SHA256:gAhxqeDkxAPTOOeKQDHBDNe/AYpARkxI8h7Rrc8pLD8
|
||||
2026/10/04 07:07:55 [INFO] Operator email sent for tester-1/offsite_key_unlocked
|
||||
2026/10/04 07:07:56 [INFO] offsitekeys: removed 3 unpinned key line(s) from tester-1's sub-account (u629488-sub4) on the operator's request
|
||||
2026/10/04 07:07:59 [INFO] offsitekeys: audit tester-1: 0 line(s), all pinned append-only
|
||||
@@ -0,0 +1,71 @@
|
||||
## Part D on demo-hp 2026-10-04T05:14:55Z
|
||||
create storage failed: tmp-dooplex-copy: error fetching datastores - 401 Unauthorized
|
||||
(pvesm add refused: it validates with --password on argv; the entry is written to storage.cfg instead, the token already sits in /etc/pve/priv/storage/tmp-dooplex-copy.pw 0600)
|
||||
total 1
|
||||
-rw------- felhom-pbs.enc
|
||||
-rw------- felhom-pbs.pw
|
||||
entry added in 0 s
|
||||
tmp-dooplex-copy: error fetching datastores - 401 Unauthorized
|
||||
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
|
||||
tmp-dooplex-copy pbs inactive 0 0 0 0.00%
|
||||
tmp-dooplex-copy: error fetching datastores - 401 Unauthorized
|
||||
listed in 8 s
|
||||
total 2
|
||||
-rw------- felhom-pbs.enc
|
||||
-rw------- felhom-pbs.pw
|
||||
-rw------- tmp-dooplex-copy.enc
|
||||
-rw------- tmp-dooplex-copy.pw
|
||||
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
|
||||
tmp-dooplex-copy pbs active 9688301348 3345981544 5853981680 34.54%
|
||||
Volid Format Type Size VMID
|
||||
tmp-dooplex-copy:backup/ct/9201/2026-09-24T20:06:25Z pbs-ct backup 23236593218 9201
|
||||
tmp-dooplex-copy:backup/ct/9201/2026-10-01T20:15:29Z pbs-ct backup 15223817296 9201
|
||||
listed in 2 s
|
||||
Formatting '/mnt/hdd_1/images/9299/vm-9299-disk-1.raw', fmt=raw size=75161927680 preallocation=off
|
||||
Creating filesystem with 18350080 4k blocks and 4587520 inodes
|
||||
Filesystem UUID: 44f4df92-7e77-4da9-9ff0-43f79ad951d3
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
|
||||
4096000, 7962624, 11239424
|
||||
restoring 'tmp-dooplex-copy:backup/ct/9201/2026-10-01T20:15:29Z' now..
|
||||
Using encryption key from file descriptor..
|
||||
Fingerprint: dd:d1:d8:53:44:62:5e:0b
|
||||
merging backed-up and given configuration..
|
||||
Using encryption key from file descriptor..
|
||||
Fingerprint: dd:d1:d8:53:44:62:5e:0b
|
||||
restore rc=0 in 186 s
|
||||
hostname: demo-hp
|
||||
memory: 25898
|
||||
mp0: nvme-scratch:9299/vm-9299-disk-1.raw,mp=/var/lib/felhom,backup=1,size=70G
|
||||
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
|
||||
mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
|
||||
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:F8:33:C2,ip=dhcp,type=veth
|
||||
onboot: 1
|
||||
rootfs: nvme-scratch:9299/vm-9299-disk-0.raw,size=32G
|
||||
mp0: nvme-scratch:9299/vm-9299-disk-1.raw,mp=/var/lib/felhom,backup=1,size=70G
|
||||
onboot: 0
|
||||
status: stopped
|
||||
mounted in 1 s at /var/lib/lxc/9299/rootfs
|
||||
rootfs: demo-hp / PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
|
||||
/var/lib/felhom: docker lost+found sys_drive
|
||||
controller data volume: /var/lib/felhom/docker/volumes/felhom-controller-data/_data
|
||||
settings.json mtime: 2026-10-01T20:05:20Z
|
||||
controller image: gitea.dooplex.hu/admin/felhom-controller:0.286.1
|
||||
felhom data size: 14G
|
||||
## live 9201 for comparison:
|
||||
unmounted
|
||||
/opt/docker/stacks -> /opt/docker/stacks
|
||||
docker volumes: 913
|
||||
## Part D teardown 2026-10-04T05:19:41Z
|
||||
purging CT 9299 from related configurations..
|
||||
9299 destroyed in 2 s
|
||||
leftover 9299 image dirs: 0
|
||||
storage entry removed
|
||||
storage.cfg mentions: 0
|
||||
felhom-pbs.enc felhom-pbs.pw
|
||||
felhom-pbs.enc (the box key) untouched
|
||||
VMID Status Lock Name
|
||||
9201 running demo-hp
|
||||
9202 running demo-hp-scratch
|
||||
DooPlex token + ACL removed
|
||||
tokens left named demohp: 0
|
||||
@@ -0,0 +1,13 @@
|
||||
## Part E live 2026-10-04T05:08:37Z — plant a set-aside dir on tester-1 (u629488-sub4)
|
||||
home: . .. .ssh felhom-repo felhom-repo.orphaned-20261004-test
|
||||
## refusal: the LIVE repository named
|
||||
HTTP 502 registrar failed: offsitekeys: "/home/felhom-repo" is not a set-aside copy of /home/felhom-repo
|
||||
|
||||
## the request (as the box sends it)
|
||||
{"state":"pending","path":"/home/felhom-repo.orphaned-20261004-test","due_at":"2026-10-04T05:11:40Z"}
|
||||
HTTP 200
|
||||
## +70 s (before the 3-min delay): status {"state":"pending","path":"/home/felhom-repo.orphaned-20261004-test","due_at":"2026-10-04T05:11:40Z"}; dir: marker.txt
|
||||
## +220 s (after the delay): status {"state":"deleted","path":"/home/felhom-repo.orphaned-20261004-test","due_at":"2026-10-04T05:11:40Z"}; dir: /usr/bin/ls: cannot access '/home/felhom-repo.orphaned-20261004-test': No such file or directory
|
||||
home: . .. .ssh felhom-repo
|
||||
2026/10/04 07:12:02 [WARN] offsitekeys: DELETED tester-1's set-aside off-site copy /home/felhom-repo.orphaned-20261004-test (requested 2026-10-04T05:08:40Z, due 2026-10-04T05:11:40Z)
|
||||
## revert 2026-10-04T05:13:54Z: manifest commit 7a0d0c0 removed OFFSITE_ABANDON_DELAY; new pod started 05:12:51Z logs NO "OFFSITE_ABANDON_DELAY" line (= 7-day default); ArgoCD Synced at 7a0d0c0
|
||||
Reference in New Issue
Block a user