scripts/hub-db-backup: DooPlex push (02:30) + weekly restore test (Sun 04:30) of the hub DB to ep0 (R-173, R-231); 15 tests, red-proofs P1-P9; Part A/B evidence
gates / gates (push) Successful in 32s

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
2026-10-05 15:41:08 +02:00
parent 5d8922e619
commit cea8502f0b
16 changed files with 842 additions and 0 deletions
@@ -0,0 +1,10 @@
## Longhorn online expansion FAILS — 2026-10-05T12:37:44Z
LAST SEEN TYPE REASON OBJECT MESSAGE
7m34s Normal ExternalExpanding persistentvolumeclaim/hub-data waiting for an external controller to expand this PVC
6m4s Warning VolumeResizeFailed persistentvolumeclaim/hub-data resize volume "pvc-486c9809-4672-4b56-b70e-0bf01d0c3628" by resizer "driver.longhorn.io" failed: rpc error: code = DeadlineExceeded desc = volume pvc-486c9809-4672-4b56-b70e-0bf01d0c3628 expansion from existing capacity 1073741824 to requested capacity 2147483648 failed
4s Normal Resizing persistentvolumeclaim/hub-data External resizer is resizing volume pvc-486c9809-4672-4b56-b70e-0bf01d0c3628
4s Warning VolumeResizeFailed persistentvolumeclaim/hub-data resize volume "pvc-486c9809-4672-4b56-b70e-0bf01d0c3628" by resizer "driver.longhorn.io" failed: rpc error: code = DeadlineExceeded desc = volume pvc-486c9809-4672-4b56-b70e-0bf01d0c3628 expansion from existing capacity 2147483648 to requested capacity 2147483648 failed
engine pvc-486c9809-4672-4b56-b70e-0bf01d0c3628-e-0 spec=2147483648 current=1073741824
[pvc-486c9809-4672-4b56-b70e-0bf01d0c3628-e-0] time="2026-10-05T12:37:45.150634697Z" level=error msg="Failed to expand the frontend" func="controller.(*Controller).Expand.func1.1" file="control.go:332" error="device pvc-486c9809-4672-4b56-b70e-0bf01d0c3628: fail to refresh iSCSI initiator: failed to execute: /usr/bin/nsenter [nsenter --mount=/host/proc/196610/ns/mnt --net=/host/proc/196610/ns/net iscsiadm --version], output , stderr nsenter: cannot open /host/proc/196610/ns/mnt: No such file or directory: exit status 1" volume=pvc-486c9809-4672-4b56-b70e-0bf01d0c3628
996780 744880 235516 76% /data
instance-manager-703c624848c15b89cb295ca89db85615 1/1 Running 0 116d 10.42.0.229 dooplex <none> <none>
@@ -0,0 +1,100 @@
## BEFORE 2026-10-05T13:10:25Z
pvc-0113bbbf-fd5c-441b-9d58-6943ffa8c133 kisfenyo-filebrowser-data kisfenyo-system attached healthy 524288000
pvc-0267c9d6-07e4-476c-8866-0731dcd85fff romm-resources arcade-system attached healthy 10737418240
pvc-0eae2db7-47f1-477b-b9a9-9968c2d49a6d gokapi-config fileshare-system attached healthy 1073741824
pvc-11b257b2-8edc-4bf6-b846-00979feb195f zipline-data zipline-system attached healthy 10737418240
pvc-17e3b3cc-9c6a-4d70-85cc-c2ae72e0b4ec plantit-db plantit-system attached healthy 2147483648
pvc-1ae51535-0b2b-4975-bd9f-e651cc7a1edd nextcloud-postgresql-data nextcloud-system attached healthy 5368709120
pvc-1c47727a-d5ac-4171-ac36-26c2df3d0d63 nextcloud-nextcloud nextcloud-system attached healthy 10737418240
pvc-1d4ec717-7ee5-40a0-b093-3a62fd2a81a4 authentik-media auth-system attached healthy 2147483648
pvc-202e3bf2-9e61-4c59-ae0c-43fa6a42b403 tailscale-state admin-system attached healthy 1073741824
pvc-22343573-266d-492e-bd0a-37296217b523 postgresql-2 database-system attached healthy 53687091200
pvc-22c87ddc-afd7-4520-8473-9e4bf2b6425a immich-machine-learning-ssd2 immich-system attached healthy 10737418240
pvc-2382e7bb-9514-409b-a2cc-e97c9725512a uptimekuma-data-ssd2 uptimekuma-system attached healthy 5368709120
pvc-297c20e6-4ac0-42c4-9c69-24580a7f12ee act-runner-data gitea-system attached healthy 5368709120
pvc-2e2e8c4c-4ba3-4dcf-9ed4-78acaf7ef92b sparkyfitness-uploads workout-system attached healthy 5368709120
pvc-328d0394-1ca7-47be-8300-2dc8a1740235 paperless-redis paperless-system attached healthy 1073741824
pvc-34a1540e-8b89-4c1d-8465-796bdbfb172f pms-config-plex-plex-media-server-0 mediaserver-system attached healthy 37580963840
pvc-38e8c624-5717-483b-8da1-613e703e36c8 umami-db-data felhom-system attached healthy 2147483648
pvc-3ae955c9-9968-4099-8a7b-af2bd87d9014 immich-valkey-ssd2 immich-system attached healthy 1073741824
pvc-3b23c214-ea15-40f8-aac5-61f783bf5beb plantit-uploads plantit-system attached healthy 5368709120
pvc-3d3d043a-92e7-4c34-8383-9d08deb96689 crafty-backups crafty-system attached healthy 53687091200
pvc-42695a34-d921-4007-b5e5-e40aa728e104 code-server-config code-system attached healthy 2147483648
pvc-459e6d2a-92b8-438e-a339-ff7e40089f3d pihole pihole-system attached healthy 5368709120
pvc-486c9809-4672-4b56-b70e-0bf01d0c3628 hub-data felhom-system attached healthy 2147483648
pvc-49844e6d-dbe7-427a-a021-8a36a5996478 onlyoffice-logs office-system attached healthy 2147483648
pvc-4b9796b8-4c76-4c12-b20e-bce7eb720baf outline-redis outline-system attached healthy 1073741824
pvc-4c75264d-5ca8-4298-9916-e84fe7130b3b onlyoffice-data office-system attached healthy 5368709120
pvc-4e3fd336-9f09-4251-bd90-cdc259ef739e wanderer-db wanderer-system attached healthy 5368709120
pvc-4e4aba1e-b414-4016-8895-ecdf0fb04a6b grafana-data mon-system attached healthy 8589934592
pvc-4fdd6da2-e795-4fa9-9e36-96f212c7018b audiobookshelf-config audiobookshelf-system attached healthy 5368709120
pvc-59667477-c72a-49ea-bf83-b51a672cc727 crafty-import crafty-system attached healthy 5368709120
pvc-5acabaa7-6017-4793-b45e-f1f4fd80ab97 code-server-local code-system attached healthy 5368709120
pvc-5c436be9-dbec-4b02-ac26-cb27ce341815 onlyoffice-lib office-system attached healthy 5368709120
pvc-5cdbcb85-dfb8-480b-9dc3-2e5d03628dd1 dev-jarr-postgres jarrs-system attached healthy 5368709120
pvc-5fbd3bbc-fb14-406d-954c-1a232ce313ce gitea-data gitea-system attached healthy 53687091200
pvc-63f3b34f-3ac6-41a5-8790-3a27db0b5a2b revfulop-calendar-data orsi-system attached healthy 268435456
pvc-69577be6-c9f1-40f7-8edf-89ce7cac7c5b prometheus-data-ssd1 mon-system attached healthy 32212254720
pvc-6db10505-5d7d-4997-9f7a-6df13cf847ba glance-helper-data glance-system attached healthy 209715200
pvc-6ed74c1a-3f10-4638-9baa-9330cbff980d opengist-data opengist-system attached healthy 5368709120
pvc-71cbab32-86f9-40a5-8f3b-220ad626c492 vaultwarden-data vaultwarden-system attached healthy 5368709120
pvc-724a5306-f7a4-4b05-ac28-630597b9b8e2 gokapi-data fileshare-system attached healthy 53687091200
pvc-72c77aad-fdc4-4551-a79a-7142fae94fa9 serverpackcreator-mongo-data crafty-system attached healthy 42949672960
pvc-79478678-07e4-4140-aef1-771ff763ab3f romm-config arcade-system attached healthy 1073741824
pvc-7d6dec2c-0cda-43c7-ad1d-44ffebcb8cb6 seerr-config-pvc servarr-system attached healthy 1073741824
pvc-7eb1f195-6a35-46c3-bbb1-5bab48209624 prowlarr-config-pvc servarr-system attached healthy 4294967296
pvc-7fc61dcd-52f0-45b4-97f1-9c82b28ab714 sparkyfitness-postgres workout-system attached healthy 5368709120
pvc-83701410-61f7-4d16-a8f6-158668110685 calcom-redis booking-system attached healthy 1073741824
pvc-8401df55-4668-40e3-97d2-d53e2c1090f1 tautulli-config-pvc servarr-system attached healthy 2147483648
pvc-85be87dc-3351-4459-9f82-55f8cc598a2f actualbudget-data actualbudget-system attached healthy 5368709120
pvc-86a960ec-ac7f-41a1-b595-00deac982ff5 serverpackcreator-data crafty-system attached healthy 42949672960
pvc-88b4d7b4-cb5d-48e0-871e-9e150872d0bd calibre-web-automated-config calibre-system attached healthy 10737418240
pvc-8931072a-e34e-4333-9ab5-9629b6078505 upsnap-data admin-system attached healthy 1073741824
pvc-8af636d2-2d00-46ec-9947-7b3143563c16 dev-jarr-redis jarrs-system attached healthy 1073741824
pvc-8ed5eea5-67d4-46fc-9806-6b8fdd61e42e filebrowser-files felhom-system attached healthy 1073741824
pvc-8fcb5bc5-865a-4fc3-8bac-b808b1233dbd tandoor-staticfiles tandoor-system attached healthy 1073741824
pvc-9a76846d-b1cf-4928-b950-2db5500531e1 wanderer-meilisearch wanderer-system attached healthy 5368709120
pvc-9ca6a486-40bb-46ac-ac48-bd49d4399f4b sonarr-config-pvc servarr-system attached healthy 8589934592
pvc-a9dd6c7c-19eb-4a0a-935d-d2c30545c378 romm-db arcade-system attached healthy 2147483648
pvc-b0e94be5-7cd8-4a4c-9295-901614e9eeb7 crafty-app-config crafty-system attached healthy 2147483648
pvc-b10da48e-6fcc-42ef-8414-d5d617028d68 qbittorrent-config-pvc servarr-system attached healthy 1073741824
pvc-b359d7de-1c7c-4b6a-9f6d-6776c2f876ef minecraft-tlauncher-data crafty-system attached healthy 21474836480
pvc-b8959b5c-6b88-4e65-81f2-0eabbf4a6034 privatebin-data privatebin-system attached healthy 2147483648
pvc-bd481dfa-68d6-4f7a-a634-4d3b76dbf57b recipe-importer-data tandoor-system attached healthy 134217728
pvc-be9d7813-1881-462e-8951-1bd1315f7dc5 filebrowser-db felhom-system attached healthy 104857600
pvc-beb0ad8a-755a-4152-a0af-f7367b32b19c paperless-config paperless-system attached healthy 10737418240
pvc-c4dfbc43-2c35-4ef0-9cfe-39b3dc885dae radarrkids-config-pvc servarr-system attached healthy 3221225472
pvc-c6b9ab63-24b8-46f2-b52a-06da86393de8 filebrowser-config web-system attached healthy 104857600
pvc-c8c0d0f1-c9ea-4ed3-9e22-1e254a026c49 radarr-config-pvc servarr-system attached healthy 8589934592
pvc-c911715e-56ee-4d97-836e-89efc448ad9d immich-postgres-ssd2 immich-system attached healthy 10737418240
pvc-cbfd55b0-4f44-487d-90fd-9c80b916ced7 crafty-servers crafty-system attached healthy 53687091200
pvc-d2ea5b66-d251-4467-9829-14654fab7a2f bookstack-mariadb-ssd2 bookstack-system attached healthy 5368709120
pvc-d9f09398-17cf-44b0-b3a2-9475bc773fec adventurelog-postgres adventurelog-system attached healthy 5368709120
pvc-e2151b14-2b24-4128-993a-c392c8d9706a code-server-workspace code-system attached healthy 21474836480
pvc-e3b26fdc-fd52-4fdf-9fa2-1d1ecfd029e4 termix-data termix-system attached healthy 5368709120
pvc-e57c220f-bc92-4815-89c9-0dee35eab3c3 orsi-filebrowser-config orsi-system attached healthy 104857600
pvc-e5b8f824-7720-454c-b546-a098608df24a alertmanager-data mon-system attached healthy 1073741824
pvc-ea977627-fb0a-4dee-8678-9ceceea448da bookstack-config-ssd2 bookstack-system attached healthy 5368709120
pvc-ef87a570-5d8d-45e5-9bbf-c700cbee1508 onlyoffice-db office-system attached healthy 5368709120
pvc-fc3d2f7f-6569-473c-acb4-002c613e5f18 filebrowser-data web-system attached healthy 5368709120
## settings
auto-delete-pod-when-volume-detached-unexpectedly=true
auto-salvage=true
## pods not Running/Completed
## instance managers
instance-manager-703c624848c15b89cb295ca89db85615 1/1 Running 0 116d 10.42.0.229 dooplex <none> <none>
instance-manager-703c624848c15b89cb295ca89db85615 aio running
## 2026-10-05T13:20:10Z DELETE instance-manager pod
pod "instance-manager-703c624848c15b89cb295ca89db85615" deleted
13:20:21Z instance-manager: 1/1 Running
instance-manager-703c624848c15b89cb295ca89db85615 1/1 Running 0 1s 10.42.0.235 dooplex <none> <none>
13:22:00Z not attached+healthy: 0
## AFTER 2026-10-05T13:35:41Z
volumes not attached+healthy: 0
pods not Running/Completed:
Running but not ready:
zipline: was CrashLoop on :latest=4.8.0 (prisma->drizzle refusal); pinned 4.7.0 in homelab-manifests 90f60e4+4c8ec7a; running, 4 migrations applied
## hub-data
capacity=2Gi conditions=
engine current=2147483648
2028392 745616 1266392 37% /data
@@ -0,0 +1,22 @@
## BEFORE sync 2026-10-05T12:29:57Z
pvc labels={"app":"hub","recurring-job-group.longhorn.io/default":"disabled"} request=1Gi capacity=1Gi
volume labels={"backup-target":"default","longhornvolume":"pvc-486c9809-4672-4b56-b70e-0bf01d0c3628","recurring-job-group.longhorn.io/default":"enabled","setting.longhorn.io/remove-snapshots-during-filesystem-trim":"ignored","setting.longhorn.io/replica-auto-balance":"ignored","setting.longhorn.io/snapshot-data-integrity":"ignored"} size=1073741824 robustness=healthy
image=gitea.dooplex.hu/admin/felhom-hub:0.135.0
## AFTER sync 2026-10-05T12:30:55Z
pvc labels={"app":"hub","recurring-job-group.longhorn.io/default":"enabled"} request=2Gi capacity=1Gi conditions=[{"lastProbeTime":null,"lastTransitionTime":"2026-10-05T12:30:10Z","status":"True","type":"Resizing"}]
volume labels={"backup-target":"default","longhornvolume":"pvc-486c9809-4672-4b56-b70e-0bf01d0c3628","recurring-job-group.longhorn.io/default":"enabled","setting.longhorn.io/remove-snapshots-during-filesystem-trim":"ignored","setting.longhorn.io/replica-auto-balance":"ignored","setting.longhorn.io/snapshot-data-integrity":"ignored"} size=2147483648 robustness=healthy
image=gitea.dooplex.hu/admin/felhom-hub:0.136.0
## hub log (snapshot + version lines)
2026/10/05 14:30:14 [INFO] felhom-hub 0.136.0 starting
2026/10/05 14:30:14 [INFO] Default controller-version floor: 0.120.0
2026/10/05 14:30:15 [INFO] Gitea artifact browser enabled (Day-0 version dropdowns) via http://gitea.gitea-system.svc.cluster.local:3000
2026/10/05 14:30:15 [INFO] Registry version checker started (every 6h)
2026/10/05 14:30:16 [DEBUG] Registry version check: latest = 0.296.0
2026/10/05 14:30:37 [INFO] db-snapshot: next run at 2026-10-06 02:00 CEST (in 11h29m23s)
## /data
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/longhorn/pvc-486c9809-4672-4b56-b70e-0bf01d0c3628
996780 507808 472588 52% /data
total 124836
-rw-r--r-- 1 root root 127840256 Oct 5 14:30 hub-20261005T123037Z.db.tmp
-rw-r--r-- 1 root root 1024 Oct 5 14:30 hub-20261005T123037Z.db.tmp-journal
@@ -0,0 +1,16 @@
## 2026-10-05T12:53:18Z scale hub to 0
deployment.apps/hub scaled
12:56:30Z volume state=attached
13:01:42Z attached expansionRequired=true
engine spec=2147483648 current=1073741824
30m Warning VolumeResizeFailed persistentvolumeclaim/hub-data resize volume "pvc-486c9809-4672-4b56-b70e-0bf01d0c3628" by resizer "driver.longhorn.io" failed: rpc error: code = DeadlineExceeded desc = volume pvc-486c9809-4672-4b56-b70e-0bf01d0c3628 expansion from existing capacity 1073741824 to requested capacity 2147483648 failed
2s Normal Resizing persistentvolumeclaim/hub-data External resizer is resizing volume pvc-486c9809-4672-4b56-b70e-0bf01d0c3628
2s Warning VolumeResizeFailed persistentvolumeclaim/hub-data resize volume "pvc-486c9809-4672-4b56-b70e-0bf01d0c3628" by resizer "driver.longhorn.io" failed: rpc error: code = DeadlineExceeded desc = volume pvc-486c9809-4672-4b56-b70e-0bf01d0c3628 expansion from existing capacity 2147483648 to requested capacity 2147483648 failed
## 2026-10-05T13:01:50Z volume never detached — attachment tickets:
{"volume-expansion-controller-pvc-486c9809-4672-4b56-b70e-0bf01d0c3628":{"generation":0,"id":"volume-expansion-controller-pvc-486c9809-4672-4b56-b70e-0bf01d0c3628","nodeID":"dooplex","parameters":{"disableFrontend":"false"},"type":"volume-expansion-controller"}}
## 2026-10-05T13:01:51Z scale hub to 1
deployment.apps/hub scaled
Waiting for deployment "hub" rollout to finish: 0 of 1 updated replicas are available...
deployment "hub" successfully rolled out
NAME READY STATUS RESTARTS AGE
hub-586df4748f-ppn4x 1/1 Running 0 54s
@@ -0,0 +1,44 @@
## 2026-10-05T13:37:43Z ACLs
## ep0 AFTER 2026-10-05T13:37:43Z
## prune jobs
[
{
"comment": "R-82 retention keep-last=2, server-side (box tokens are write-only)",
"id": "prune-demo-felhom",
"keep-last": 2,
"max-depth": 0,
"ns": "demo-felhom",
"schedule": "03:30",
"store": "felhom-offsite"
},
{
"comment": "R-82 retention keep-last=2, server-side (box tokens are write-only)",
"id": "prune-demo-hp",
"keep-last": 2,
"max-depth": 0,
"ns": "demo-hp",
"schedule": "03:30",
"store": "felhom-offsite"
},
{
"comment": "R-173 hub DB copies; operator ns only",
"id": "prune-operator-hubdb",
"keep-daily": 14,
"keep-weekly": 8,
"max-depth": 0,
"ns": "operator",
"schedule": "03:45",
"store": "felhom-offsite"
}
]
## gc
felhom-offsite sun 04:30
## users
['felhom@pbs', 'dooplex-hub@pbs', 'root@pam']
## tokens (ids only)
['dooplex-hub@pbs!restore', 'dooplex-hub@pbs!push']
## ACLs naming dooplex-hub
/datastore/felhom-offsite/operator dooplex-hub@pbs!restore DatastoreReader True
/datastore/felhom-offsite/operator dooplex-hub@pbs DatastoreReader True
/datastore/felhom-offsite/operator dooplex-hub@pbs DatastoreBackup True
/datastore/felhom-offsite/operator dooplex-hub@pbs!push DatastoreBackup True
@@ -0,0 +1,53 @@
## ep0 BEFORE 2026-10-05T13:36:16Z host=felhom-hetzner
proxmox-backup-server 4.2.8-1 running version: 4.2.5
## prune jobs
[
{
"comment": "R-82 retention keep-last=2, server-side (box tokens are write-only)",
"id": "prune-demo-hp",
"keep-last": 2,
"max-depth": 0,
"ns": "demo-hp",
"schedule": "03:30",
"store": "felhom-offsite"
},
{
"comment": "R-82 retention keep-last=2, server-side (box tokens are write-only)",
"id": "prune-demo-felhom",
"keep-last": 2,
"max-depth": 0,
"ns": "demo-felhom",
"schedule": "03:30",
"store": "felhom-offsite"
}
]
## garbage-collection
[
{
"cache-stats": {
"hits": 6726,
"misses": 8569
},
"disk-bytes": 11794377956,
"disk-chunks": 8570,
"duration": 44,
"index-data-bytes": 51257596032,
"index-file-count": 8,
"last-run-endtime": 1791088244,
"last-run-state": "OK",
"next-run": 1791693000,
"pending-bytes": 0,
"pending-chunks": 0,
"removed-bad": 0,
"removed-bytes": 8871579815,
"removed-chunks": 7465,
"schedule": "sun 04:30",
"still-bad": 0,
"store": "felhom-offsite",
"upid": "UPID:felhom-hetzner:00086AE7:07B20B30:000002A6:6AC1D648:garbage_collection:felhom\\x2doffsite:root@pam:"
}
]
## users (id only)
['root@pam', 'felhom@pbs']
## operator ns exists?
False
@@ -0,0 +1,46 @@
## 2026-10-05T13:37:04Z changes
user created
thread 'main' (2343121) panicked at /usr/share/cargo/registry/proxmox-router-3.2.8/src/cli/text_table.rs:812:13:
not implemented
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: parameter verification failed - 'schedule': unable to parse calendar event at 'daily' - Context("weekday")
Usage: proxmox-backup-manager prune-job create <id> --schedule <calendar-event> --store <string> [OPTIONS]
<id> <string>
Job ID.
--schedule <calendar-event>
Run prune job at specified schedule.
--store <string>
Datastore name.
Optional parameters:
--comment <string>
Comment.
--disable <boolean> (default=false)
Disable this job.
--keep-daily <integer> (1 - N)
Number of daily backups to keep.
--keep-hourly <integer> (1 - N)
Number of hourly backups to keep.
--keep-last <integer> (1 - N)
Number of backups to keep.
--keep-monthly <integer> (1 - N)
Number of monthly backups to keep.
--keep-weekly <integer> (1 - N)
Number of weekly backups to keep.
--keep-yearly <integer> (1 - N)
Number of yearly backups to keep.
--max-depth <integer> (0 - 7)
How many levels of namespaces should be operated on (0 == no
recursion, empty == automatic full recursion, namespace depths
reduce maximum allowed value)
--ns <string>
Namespace.
## 2026-10-05T13:37:12Z check + retry
operator ns exists: True
Prune job created: prune-operator-hubdb
prune job created
@@ -0,0 +1,76 @@
### P1 push: integrity_check ignored
test_corrupt_copy_is_never_pushed (__main__.Push.test_corrupt_copy_is_never_pushed) ... FAIL
FAIL: test_corrupt_copy_is_never_pushed (__main__.Push.test_corrupt_copy_is_never_pushed)
AssertionError: 0 == 0
Ran 1 test in 0.205s
FAILED (failures=1)
### P2 push: snapshot age ignored
test_stale_snapshot_is_not_pushed_again (__main__.Push.test_stale_snapshot_is_not_pushed_again) ... FAIL
FAIL: test_stale_snapshot_is_not_pushed_again (__main__.Push.test_stale_snapshot_is_not_pushed_again)
AssertionError: 0 == 0
Ran 1 test in 0.201s
FAILED (failures=1)
### P3 push: success signal after a failed push
test_failed_push_writes_no_signal (__main__.Push.test_failed_push_writes_no_signal) ... FAIL
FAIL: test_failed_push_writes_no_signal (__main__.Push.test_failed_push_writes_no_signal)
AssertionError: 0 == 0
Ran 1 test in 0.206s
FAILED (failures=1)
### P4 push: size check removed
test_truncated_copy_is_not_pushed (__main__.Push.test_truncated_copy_is_not_pushed) ... ok
Ran 1 test in 0.156s
OK
### P5 push: no encryption flag
test_happy_path_pushes_encrypted_to_operator_and_writes_signal (__main__.Push.test_happy_path_pushes_encrypted_to_operator_and_writes_signal) ... ERROR
ERROR: test_happy_path_pushes_encrypted_to_operator_and_writes_signal (__main__.Push.test_happy_path_pushes_encrypted_to_operator_and_writes_signal)
Ran 1 test in 0.211s
FAILED (errors=1)
### P6 restore: readable console passwords allowed
test_readable_console_password_fails (__main__.RestoreTest.test_readable_console_password_fails) ... FAIL
FAIL: test_readable_console_password_fails (__main__.RestoreTest.test_readable_console_password_fails)
AssertionError: 0 == 0
Ran 1 test in 0.357s
FAILED (failures=1)
### P7 restore: uses the push token
test_happy_path_writes_signal_with_the_read_only_token (__main__.RestoreTest.test_happy_path_writes_signal_with_the_read_only_token) ... FAIL
FAIL: test_happy_path_writes_signal_with_the_read_only_token (__main__.RestoreTest.test_happy_path_writes_signal_with_the_read_only_token)
AssertionError: False is not true : {'argv': ['snapshot', 'list', 'host/dooplex-hub', '--ns', 'operator', '--output-format', 'json', '--repository', 'dooplex-hub@pbs!restore@127.0.0.1:18007:felhom-offsite'], 'pw': '/tmp/tmp0xabeyiz/conf/token-push', 'fp': 'aa:bb'}
Ran 1 test in 0.371s
FAILED (failures=1)
### P8 restore: copy age ignored
test_old_copy_fails (__main__.RestoreTest.test_old_copy_fails) ... FAIL
FAIL: test_old_copy_fails (__main__.RestoreTest.test_old_copy_fails)
AssertionError: 0 == 0
Ran 1 test in 0.352s
FAILED (failures=1)
### P9 push: staged plaintext left behind
test_happy_path_pushes_encrypted_to_operator_and_writes_signal (__main__.Push.test_happy_path_pushes_encrypted_to_operator_and_writes_signal) ... FAIL
FAIL: test_happy_path_pushes_encrypted_to_operator_and_writes_signal (__main__.Push.test_happy_path_pushes_encrypted_to_operator_and_writes_signal)
AssertionError: Lists differ: ['hub.db'] != []
Ran 1 test in 0.200s
FAILED (failures=1)
P4 and P5 above: P4 did NOT convict (masked by integrity_check), P5 ERRORED rather than failed. Tests strengthened; re-runs:
### P4 (re-run) push: size check removed
test_truncated_copy_is_not_pushed (__main__.Push.test_truncated_copy_is_not_pushed) ... FAIL
FAIL: test_truncated_copy_is_not_pushed (__main__.Push.test_truncated_copy_is_not_pushed)
AssertionError: "bytes, the pod's file is" not found in 'felhom-hub-db-backup: FAILED: integrity_check: Error: in prepare, database disk image is malformed (11)\n'
Ran 1 test in 0.146s
FAILED (failures=1)
### P5 (re-run) push: no encryption flag
test_happy_path_pushes_encrypted_to_operator_and_writes_signal (__main__.Push.test_happy_path_pushes_encrypted_to_operator_and_writes_signal) ... FAIL
FAIL: test_happy_path_pushes_encrypted_to_operator_and_writes_signal (__main__.Push.test_happy_path_pushes_encrypted_to_operator_and_writes_signal)
AssertionError: '--crypt-mode' not found in ['backup', 'hubdb.pxar:/tmp/tmpq980kr66/state/stage', '--ns', 'operator', '--backup-type', 'host', '--backup-id', 'dooplex-hub', '--keyfile', '/tmp/tmpq980kr66/conf/enc.key', '--repository', 'dooplex-hub@pbs!push@127.0.0.1:18007:felhom-offsite'] : push without --crypt-mode
Ran 1 test in 0.200s
FAILED (failures=1)