DRILL R-95: the recovery route does not exist — stopped before the destructive phase
gates / gates (push) Successful in 19s
gates / gates (push) Successful in 19s
The drill was to delete demo-hp's off-site history and get it back out of a Storage
Box snapshot, filling row 10's blank RTO. Phase 1 found there is nothing to get it
back from: no snapshot is reachable from a sub-account BY ANY NAME.
Measured, read-only, no delete verb issued against any live store:
- 777,600 exact names in the vendor form YYYY-MM-DDTHH-MM-SS, nine full days at
second granularity, plus 126 alternative shapes -> ZERO hits.
- The control is what makes that mean anything: the identical 600-name batch shape
with one real path appended returned it, 6 of 6.
- Structural cause: /home (u629488-sub3) is st_dev 0,82; /.zfs/snapshot is st_dev
0,276; /home/.zfs does not exist. A snapshot under /.zfs/snapshot belongs to a
different dataset than the one holding felhom-repo.
- Three tools agree with controls in the same run: SFTP, the port-23 shell,
rsync --list-only.
So yesterday's re-scope splits: clause (a) "the box cannot write into the snapshot
area" STANDS and is re-confirmed; clause (b) "the rest is recoverable file by file"
is NOT SUPPORTED. STOPPED before Phase 2 on the operator's ruling — with no recovery
leg the deletion would have destroyed real history to buy only an alarm test that
could not fire at the specified size. Store verified untouched at 69 snapshots.
R-432 ANSWERED (negatively; its panel-read next step withdrawn as unnecessary).
R-433 no snapshot reachable by any name — decides R-95's remedy and its rank.
R-434 the drop alarm's text promises a file-by-file recovery that cannot be performed.
R-435 the drop detector is blind to a single-app deletion (>50% of 69 needed, ~9 given).
R-436 LEAD: the provider offers `rclone serve restic --stdio` and restic 0.14.0 speaks
`rclone:` (measured, controlled) — real prevention may need no new machine, IF
the vendor pins --append-only. Ask before building.
07 §8 row 10: text corrected, status NOT moved, RTO still blank.
No code, no version bump, no image, no golden. REPORT.md's only copy of the R-331
report preserved as REPORT-r331-backup-card.md before overwrite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM
This commit is contained in:
@@ -0,0 +1,52 @@
|
||||
# Evidence — DRILL: the deletion we said is survivable (R-95 recovery drill, 2026-09-01)
|
||||
|
||||
**The drill STOPPED at the end of Phase 1, on the operator's ruling, before any destructive step.**
|
||||
**No delete verb was issued against any live store. No byte on either Storage Box sub-account was
|
||||
written, moved or removed.** Every probe in this directory is a read: `ls`, `stat`, `tree`, `du`,
|
||||
`df`, `rsync --list-only`, `restic snapshots`.
|
||||
|
||||
## Why it stopped
|
||||
|
||||
Phase 1 was supposed to *find* a Storage Box snapshot so Phase 4 could recover from it. It found
|
||||
that **no snapshot is reachable from the box by any name**, and that **no unfenced route to one
|
||||
exists for CC at all**. The recovery leg therefore could not run, and deleting a real app's off-site
|
||||
history would have bought only an alarm test that — measured against the shipped threshold — could
|
||||
not have fired at the size the runbook specifies. Put to the operator as a two-option decision;
|
||||
the ruling was **stop and report**.
|
||||
|
||||
## Files
|
||||
|
||||
| file | what it is |
|
||||
|---|---|
|
||||
| `phase0-ground-truth/01-offsite-inventory.txt` | demo-hp's full off-site inventory — 69 restic snapshots, 9 apps, ids + tags + paths. The baseline. |
|
||||
| `phase0-ground-truth/02-hub-view.txt` | the hub's latest `offsite` object for demo-hp (`snapshot_count: 69`, `stats_known: true`, `last_status: ok`). Cross-checks the restic count exactly. |
|
||||
| `phase1-find-the-snapshot/01-probe-tree.txt` | SFTP probes of `/.zfs`, `/.zfs/snapshot`, `/home/.zfs`, with a positive and a negative control. Reproduces R-432. |
|
||||
| `phase1-find-the-snapshot/02-shell-and-rsync-doors.txt` | two independent tools (a real shell on port 23, and `rsync --list-only`) agree the tree lists empty. Both controlled. |
|
||||
| `phase1-find-the-snapshot/03-shell-capabilities.txt` | the port-23 restricted shell's full `help` — the command set the credential actually reaches. |
|
||||
| `phase1-find-the-snapshot/04-enumeration-attempts.txt` | `df`/`stat`/`tree`/`du` on the snapshot door. **Where the st_dev split was found.** |
|
||||
| `phase1-find-the-snapshot/05-home-zfs-door.txt` | `/home/.zfs` does not exist — three tools, negative control in the same run. |
|
||||
| `phase1-find-the-snapshot/06-name-sweep.txt` | the name sweep: **nine full days, second granularity, 777,600 candidate names, zero hits.** |
|
||||
| `phase1-find-the-snapshot/07-sweep-control.txt` | **the control for that sweep** — the identical 600-name batch shape with one real path appended; 6/6 batches returned it. Without this the zero result would prove nothing. |
|
||||
| `phase1-find-the-snapshot/08-alt-formats.txt` | 126 non-timestamp name shapes and alternative snapshot paths. Only the controls resolved. |
|
||||
| `phase1-find-the-snapshot/09-rclone-restic-lead.txt` | the `rclone:` backend measurement behind R-436, with the `banana:` invalid-backend control. |
|
||||
| `phase6-teardown/01-verify-then-clean.txt` | the store is untouched: still 69 snapshots, home is exactly `.ssh` + `felhom-repo`, repo top level intact. |
|
||||
| `phase6-teardown/02-guest-and-host-clean.txt` | scratch removed from the guest and the PVE host; controller 0.232.0 healthy, 19 healthy containers. |
|
||||
| `phase6-teardown/03-dooplex-clean-and-final-state.txt` | hub DB copies shredded (both, `-wal` included); both customers' tiers still `ok`; no alarm event raised by this session. |
|
||||
|
||||
## The method that makes the negative result trustworthy
|
||||
|
||||
A sweep that finds nothing is worthless unless it is shown it *could* have found something. The
|
||||
oracle is a batched `stat -c %n` over the port-23 shell: 500–600 paths per round trip, and stdout
|
||||
carries **only the paths that exist**. `07-sweep-control.txt` runs the identical batch shape with
|
||||
`/home` appended and gets `/home` back, 6 times out of 6. So the zero in `06-name-sweep.txt` is a
|
||||
measurement, not a silence.
|
||||
|
||||
## Not done, and why
|
||||
|
||||
* **No Hetzner API call.** Fenced by the runbook (§11-D). The hub's own API client has no snapshot
|
||||
method at all, so even unfenced it would have needed new code.
|
||||
* **No panel action.** No browser on DooPlex, and "Restore snapshot" is fenced in any case — it rolls
|
||||
back the whole Storage Box and deletes newer snapshots.
|
||||
* **`demo-felhom` never addressed.** Every storage-box connection in this session authenticated as
|
||||
`u629488-sub3`, demo-hp's own sub-account. Its tier is verified still reporting `ok` in
|
||||
`phase6-teardown/03-*`.
|
||||
+162
@@ -0,0 +1,162 @@
|
||||
### date (guest UTC): 2026-09-01T14:30:31Z
|
||||
### restic snapshots --json is too noisy; table form:
|
||||
ID Time Host Tags Paths
|
||||
------------------------------------------------------------------------------------------------------------------------------
|
||||
41c830db 2026-08-09 08:30:38 demo-hp felhom-offbox,calibre-web /mnt/sys_drive/felhom-data/backups/primary/calibre-web
|
||||
/mnt/sys_drive/felhom-data/userdata/media/books
|
||||
|
||||
9e38b84c 2026-08-09 08:30:49 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
78b93f04 2026-08-09 08:30:53 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
f06ad130 2026-08-23 02:16:25 demo-hp felhom-offbox,bookstack /mnt/sys_drive/felhom-data/backups/primary/bookstack
|
||||
|
||||
e68f6444 2026-08-23 02:16:32 demo-hp felhom-offbox,calibre-web /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
|
||||
/mnt/felhom-drives/hdd_1/userdata/media/books
|
||||
|
||||
d1a614b5 2026-08-23 02:16:36 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
e23e34a5 2026-08-23 02:16:39 demo-hp felhom-offbox,docmost /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
|
||||
5c3492ad 2026-08-23 02:16:45 demo-hp felhom-offbox,kimai /mnt/sys_drive/felhom-data/backups/primary/kimai
|
||||
|
||||
82ef8692 2026-08-23 02:16:55 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
778e8feb 2026-08-23 02:16:58 demo-hp felhom-offbox,paperless-ngx /mnt/felhom-drives/hdd_1/appdata/paperless/media
|
||||
/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx
|
||||
|
||||
8cc0e4b9 2026-08-23 02:17:04 demo-hp felhom-offbox,romm /mnt/felhom-drives/hdd_1/backups/primary/romm
|
||||
|
||||
936a30c3 2026-08-26 02:16:29 demo-hp felhom-offbox,kimai /mnt/sys_drive/felhom-data/backups/primary/kimai
|
||||
|
||||
13c3cf9d 2026-08-26 02:16:39 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
9e3474ec 2026-08-26 02:16:43 demo-hp felhom-offbox,romm /mnt/felhom-drives/hdd_1/backups/primary/romm
|
||||
|
||||
a2aec191 2026-08-26 02:16:49 demo-hp felhom-offbox,paperless-ngx /mnt/felhom-drives/hdd_1/appdata/paperless/media
|
||||
/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx
|
||||
|
||||
389a3a6d 2026-08-26 02:16:55 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
c7408e7d 2026-08-26 02:16:59 demo-hp felhom-offbox,bookstack /mnt/sys_drive/felhom-data/backups/primary/bookstack
|
||||
|
||||
f40c8fd9 2026-08-26 02:17:03 demo-hp felhom-offbox,calibre-web /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
|
||||
/mnt/felhom-drives/hdd_1/userdata/media/books
|
||||
|
||||
ea5e9a25 2026-08-26 02:17:07 demo-hp felhom-offbox,docmost /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
|
||||
e6ee92ac 2026-08-27 02:16:26 demo-hp felhom-offbox,paperless-ngx /mnt/felhom-drives/hdd_1/appdata/paperless/media
|
||||
/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx
|
||||
|
||||
73266947 2026-08-27 02:16:34 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
794d8219 2026-08-27 02:16:37 demo-hp felhom-offbox,bookstack /mnt/sys_drive/felhom-data/backups/primary/bookstack
|
||||
|
||||
b1e4d6a5 2026-08-27 02:16:41 demo-hp felhom-offbox,calibre-web /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
|
||||
/mnt/felhom-drives/hdd_1/userdata/media/books
|
||||
|
||||
7eea66d2 2026-08-27 02:16:45 demo-hp felhom-offbox,docmost /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
|
||||
b9db5519 2026-08-27 02:16:50 demo-hp felhom-offbox,kimai /mnt/sys_drive/felhom-data/backups/primary/kimai
|
||||
|
||||
0751dad2 2026-08-27 02:17:00 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
3714731c 2026-08-27 02:17:03 demo-hp felhom-offbox,romm /mnt/felhom-drives/hdd_1/backups/primary/romm
|
||||
|
||||
8dcbd53a 2026-08-28 02:16:30 demo-hp felhom-offbox,bookstack /mnt/sys_drive/felhom-data/backups/primary/bookstack
|
||||
|
||||
5a635785 2026-08-28 02:16:34 demo-hp felhom-offbox,calibre-web /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
|
||||
/mnt/felhom-drives/hdd_1/userdata/media/books
|
||||
|
||||
035e923f 2026-08-28 02:16:38 demo-hp felhom-offbox,docmost /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
|
||||
6f613bbd 2026-08-28 02:16:43 demo-hp felhom-offbox,kimai /mnt/sys_drive/felhom-data/backups/primary/kimai
|
||||
|
||||
f0c6ce70 2026-08-28 02:16:53 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
02f1de9d 2026-08-28 02:16:56 demo-hp felhom-offbox,romm /mnt/felhom-drives/hdd_1/backups/primary/romm
|
||||
|
||||
726c1ce8 2026-08-28 02:17:02 demo-hp felhom-offbox,paperless-ngx /mnt/felhom-drives/hdd_1/appdata/paperless/media
|
||||
/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx
|
||||
|
||||
c0e9dc25 2026-08-28 02:17:08 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
0b90da52 2026-08-29 02:16:26 demo-hp felhom-offbox,bookstack /mnt/sys_drive/felhom-data/backups/primary/bookstack
|
||||
|
||||
6aee1818 2026-08-29 02:16:30 demo-hp felhom-offbox,calibre-web /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
|
||||
/mnt/felhom-drives/hdd_1/userdata/media/books
|
||||
|
||||
7aea3e6d 2026-08-29 02:16:34 demo-hp felhom-offbox,docmost /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
|
||||
2f043b35 2026-08-29 02:16:39 demo-hp felhom-offbox,kimai /mnt/sys_drive/felhom-data/backups/primary/kimai
|
||||
|
||||
fa862b02 2026-08-29 02:16:49 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
3f93ec62 2026-08-29 02:16:52 demo-hp felhom-offbox,romm /mnt/felhom-drives/hdd_1/backups/primary/romm
|
||||
|
||||
ebade956 2026-08-29 02:16:58 demo-hp felhom-offbox,paperless-ngx /mnt/felhom-drives/hdd_1/appdata/paperless/media
|
||||
/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx
|
||||
|
||||
983a0365 2026-08-29 02:17:06 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
9ae0aa88 2026-08-30 02:16:29 demo-hp felhom-offbox,romm /mnt/felhom-drives/hdd_1/backups/primary/romm
|
||||
|
||||
2404511d 2026-08-30 02:16:35 demo-hp felhom-offbox,paperless-ngx /mnt/felhom-drives/hdd_1/appdata/paperless/media
|
||||
/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx
|
||||
|
||||
4a5db84d 2026-08-30 02:16:42 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
ea40c93a 2026-08-30 02:16:45 demo-hp felhom-offbox,bookstack /mnt/sys_drive/felhom-data/backups/primary/bookstack
|
||||
|
||||
37b7b51a 2026-08-30 02:16:49 demo-hp felhom-offbox,calibre-web /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
|
||||
/mnt/felhom-drives/hdd_1/userdata/media/books
|
||||
|
||||
ee67d85d 2026-08-30 02:16:53 demo-hp felhom-offbox,docmost /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
|
||||
84542ec8 2026-08-30 02:16:58 demo-hp felhom-offbox,kimai /mnt/sys_drive/felhom-data/backups/primary/kimai
|
||||
|
||||
a5c5c973 2026-08-30 02:17:08 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
05c3346a 2026-08-31 21:29:03 demo-hp felhom-offbox,bookstack /mnt/sys_drive/felhom-data/backups/primary/bookstack
|
||||
|
||||
b6b7c205 2026-08-31 21:29:08 demo-hp felhom-offbox,calibre-web /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
|
||||
/mnt/felhom-drives/hdd_1/userdata/media/books
|
||||
|
||||
0b5781f5 2026-08-31 21:29:12 demo-hp felhom-offbox,docmost /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
|
||||
6a2672b0 2026-08-31 21:29:16 demo-hp felhom-offbox,paperless-ngx /mnt/felhom-drives/hdd_1/appdata/paperless/media
|
||||
/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx
|
||||
|
||||
d3970ee3 2026-08-31 21:29:21 demo-hp felhom-offbox,kimai /mnt/sys_drive/felhom-data/backups/primary/kimai
|
||||
|
||||
5d5bee7f 2026-08-31 21:29:31 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
7b0a7161 2026-08-31 21:29:34 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
1b8c7361 2026-08-31 21:29:38 demo-hp felhom-offbox,romm /mnt/felhom-drives/hdd_1/backups/primary/romm
|
||||
|
||||
5dcee6e3 2026-08-31 21:29:44 demo-hp felhom-offbox,bentopdf /mnt/sys_drive/felhom-data/backups/primary/bentopdf
|
||||
|
||||
6fee3b5a 2026-09-01 02:16:30 demo-hp felhom-offbox,calibre-web /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
|
||||
/mnt/felhom-drives/hdd_1/userdata/media/books
|
||||
|
||||
b2e059ee 2026-09-01 02:16:34 demo-hp felhom-offbox,docmost /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
|
||||
1e179cfe 2026-09-01 02:16:39 demo-hp felhom-offbox,paperless-ngx /mnt/felhom-drives/hdd_1/appdata/paperless/media
|
||||
/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx
|
||||
|
||||
f69e0510 2026-09-01 02:16:45 demo-hp felhom-offbox,kimai /mnt/sys_drive/felhom-data/backups/primary/kimai
|
||||
|
||||
8b9d9a94 2026-09-01 02:16:55 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
1e548d3b 2026-09-01 02:16:58 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
ae903b66 2026-09-01 02:17:01 demo-hp felhom-offbox,romm /mnt/felhom-drives/hdd_1/backups/primary/romm
|
||||
|
||||
9d002b38 2026-09-01 02:17:08 demo-hp felhom-offbox,bentopdf /mnt/sys_drive/felhom-data/backups/primary/bentopdf
|
||||
|
||||
7a855a65 2026-09-01 02:17:11 demo-hp felhom-offbox,bookstack /mnt/sys_drive/felhom-data/backups/primary/bookstack
|
||||
------------------------------------------------------------------------------------------------------------------------------
|
||||
69 snapshots
|
||||
|
||||
### rc=0
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
report id 22221 received_at 2026-09-01 14:45:18
|
||||
top-level keys: ['app_telemetry', 'backup', 'claimed', 'config_hash', 'containers', 'controller_url', 'controller_version', 'customer_id', 'customer_name', 'dr_recipe', 'geo_restriction', 'health', 'offsite', 'stacks', 'storage', 'system', 'timestamp', 'version']
|
||||
--- offsite ---
|
||||
{
|
||||
"enabled": true,
|
||||
"escrow_state": "escrowed",
|
||||
"last_run": "2026-09-01T02:17:57Z",
|
||||
"last_status": "ok",
|
||||
"last_success": "2026-09-01T02:17:57Z",
|
||||
"snapshot_count": 69,
|
||||
"repo_size_bytes": 147274432,
|
||||
"quota_gb": 50,
|
||||
"stats_known": true,
|
||||
"last_integrity_check": "2026-09-01T08:24:24Z",
|
||||
"last_integrity_ok": true,
|
||||
"last_integrity_depth": "100%",
|
||||
"last_proof_run": "2026-09-01T04:04:48Z",
|
||||
"last_proof_stack": "bookstack",
|
||||
"last_proof_snapshot": "7a855a65",
|
||||
"last_proof_result": "pass"
|
||||
}
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
--- ls -la .
|
||||
Connected to u629488-sub3.your-storagebox.de.
|
||||
sftp> ls -la .
|
||||
drwxr-xr-x ? u629488-sub3 1058 4 Sep 1 12:12 ./.
|
||||
dr-x--x--x ? root root 11 Jul 21 16:01 ./..
|
||||
drwx------ ? u629488-sub3 1058 3 Jul 23 09:53 ./.ssh
|
||||
drwxrwxr-x ? u629488-sub3 1058 8 Aug 4 12:38 ./felhom-repo
|
||||
--- ls -la ./zzz-no-such-r95-negative-control
|
||||
Connected to u629488-sub3.your-storagebox.de.
|
||||
sftp> ls -la ./zzz-no-such-r95-negative-control
|
||||
Can't ls: "/home/./zzz-no-such-r95-negative-control" not found
|
||||
--- ls -la /.zfs
|
||||
Connected to u629488-sub3.your-storagebox.de.
|
||||
sftp> ls -la /.zfs
|
||||
drwxrwxrwx ? root root 0 Jul 21 16:01 /.zfs/.
|
||||
dr-x--x--x ? root root 11 Jul 21 16:01 /.zfs/..
|
||||
drwxrwxrwx ? root root 2 Sep 1 12:12 /.zfs/shares
|
||||
drwxrwxrwx ? root root 2 Jan 1 1970 /.zfs/snapshot
|
||||
--- ls -la /.zfs/snapshot
|
||||
Connected to u629488-sub3.your-storagebox.de.
|
||||
sftp> ls -la /.zfs/snapshot
|
||||
drwxrwxrwx ? root root 2 Jan 1 1970 /.zfs/snapshot/.
|
||||
drwxrwxrwx ? root root 0 Jul 21 16:01 /.zfs/snapshot/..
|
||||
--- ls -la /home/.zfs
|
||||
Connected to u629488-sub3.your-storagebox.de.
|
||||
sftp> ls -la /home/.zfs
|
||||
Can't ls: "/home/.zfs" not found
|
||||
--- ls -la /home/.zfs/snapshot
|
||||
Connected to u629488-sub3.your-storagebox.de.
|
||||
sftp> ls -la /home/.zfs/snapshot
|
||||
Can't ls: "/home/.zfs/snapshot" not found
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
=== A. ssh with a command (limited shell?) ===
|
||||
total 0
|
||||
drwxrwxrwx 2 root root 2 Jan 1 1970 .
|
||||
drwxrwxrwx 1 root root 0 Jul 21 16:01 ..
|
||||
rc=0
|
||||
|
||||
=== B. rsync --list-only on the account home (POSITIVE CONTROL) ===
|
||||
drwxr-xr-x 4 2026/09/01 12:12:48 .
|
||||
drwx------ 3 2026/07/23 09:53:43 .ssh
|
||||
drwxrwxr-x 8 2026/08/04 12:38:16 felhom-repo
|
||||
rc=0
|
||||
|
||||
=== C. rsync --list-only on /.zfs/snapshot/ (THE PROBE) ===
|
||||
drwxrwxrwx 2 1970/01/01 00:00:00 .
|
||||
rc=0
|
||||
|
||||
=== D. rsync --list-only on a path that must NOT exist (NEGATIVE CONTROL) ===
|
||||
rsync: [sender] change_dir "/zzz-no-such-r95" failed: No such file or directory (2)
|
||||
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1874) [Receiver=3.2.7]
|
||||
rsync: [Receiver] write error: Broken pipe (32)
|
||||
rc=0
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
=== id / pwd / shell ===
|
||||
Command not found. Use 'help' to get a list of available commands.
|
||||
=== ls / ===
|
||||
/usr/bin/ls: cannot open directory '/': Permission denied
|
||||
=== mount ===
|
||||
/usr/bin/cat: /proc/self/mountinfo: No such file or directory
|
||||
=== which tools ===
|
||||
Command not found. Use 'help' to get a list of available commands.
|
||||
=== help ===
|
||||
+------------------------------------------------------------------------------+
|
||||
| The following commands are available: |
|
||||
| ls|ll list directory content |
|
||||
| tree list directory content |
|
||||
| cd change current working directory |
|
||||
| pwd show current working directory |
|
||||
| mkdir create new directory |
|
||||
| rmdir delete directory |
|
||||
| du disk usage of files/directories |
|
||||
| df show disk usage |
|
||||
| dd read and write files |
|
||||
| cat output file content |
|
||||
| touch create new file |
|
||||
| cp copy files/directories |
|
||||
| rm delete files/directories |
|
||||
| unlink delete file/directory |
|
||||
| mv move files/directories |
|
||||
| chmod change file/directory permissions |
|
||||
| md5|sha1|sha256|sha512 create hash sum of file |
|
||||
| md5sum|sha1sum|sha256sum|sha512sum create hash sum of file |
|
||||
| head show first lines of file |
|
||||
| tail show last lines of file |
|
||||
| grep search for specific string in files |
|
||||
| stat stat files/directory |
|
||||
| version show version of this SSH environment |
|
||||
| |
|
||||
| Available as server side backend: |
|
||||
| borg |
|
||||
| rsync |
|
||||
| scp |
|
||||
| sftp |
|
||||
| rclone serve restic --stdio |
|
||||
| |
|
||||
| Please note that this is only a restricted shell which do not |
|
||||
| support shell features like redirects or pipes. |
|
||||
| |
|
||||
| You can find more information in our Docs: |
|
||||
| https://docs.hetzner.com/storage/storage-box/ |
|
||||
+------------------------------------------------------------------------------+
|
||||
=== snapshot-ish commands ===
|
||||
Command not found. Use 'help' to get a list of available commands.
|
||||
--
|
||||
Command not found. Use 'help' to get a list of available commands.
|
||||
--
|
||||
Command not found. Use 'help' to get a list of available commands.
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
=== df ===
|
||||
Filesystem 1K-blocks Used Available Use% Mounted on
|
||||
u629488-sub3 1073645568 2685568 1070960000 1% /home
|
||||
=== stat /.zfs/snapshot ===
|
||||
File: /.zfs/snapshot
|
||||
Size: 2 Blocks: 0 IO Block: 512 directory
|
||||
Device: 0,276 Inode: 281474976710653 Links: 2
|
||||
Access: (0777/drwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
|
||||
Access: 2026-09-01 14:36:07.861516179 +0000
|
||||
Modify: 1970-01-01 00:00:00.000000000 +0000
|
||||
Change: 1970-01-01 00:00:00.000000000 +0000
|
||||
Birth: -
|
||||
=== ls -a /.zfs/snapshot ===
|
||||
.
|
||||
..
|
||||
=== tree /.zfs ===
|
||||
/.zfs
|
||||
├── shares
|
||||
└── snapshot
|
||||
|
||||
3 directories, 0 files
|
||||
=== du /.zfs/snapshot ===
|
||||
0 /.zfs/snapshot
|
||||
=== multi-arg stat test (2 real, 1 fake) ===
|
||||
File: /home
|
||||
Size: 4 Blocks: 1 IO Block: 131072 directory
|
||||
Device: 0,82 Inode: 2273 Links: 4
|
||||
Access: (0755/drwxr-xr-x) Uid: ( 1057/u629488-sub3) Gid: ( 1058/ UNKNOWN)
|
||||
Access: 2026-07-21 16:01:41.810597438 +0000
|
||||
Modify: 2026-09-01 12:12:48.637707029 +0000
|
||||
Change: 2026-09-01 12:12:48.637707029 +0000
|
||||
Birth: 2026-07-21 16:01:41.810597438 +0000
|
||||
File: /.zfs/snapshot
|
||||
Size: 2 Blocks: 0 IO Block: 512 directory
|
||||
Device: 0,276 Inode: 281474976710653 Links: 2
|
||||
Access: (0777/drwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
|
||||
Access: 2026-09-01 14:36:10.597470894 +0000
|
||||
Modify: 1970-01-01 00:00:00.000000000 +0000
|
||||
Change: 1970-01-01 00:00:00.000000000 +0000
|
||||
Birth: -
|
||||
/usr/bin/stat: cannot statx '/zzz-no-such-r95': No such file or directory
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
=== stat /home/.zfs and /home/.zfs/snapshot (fake path = negative control) ===
|
||||
/usr/bin/stat: cannot statx '/home/.zfs': No such file or directory
|
||||
/usr/bin/stat: cannot statx '/home/.zfs/snapshot': No such file or directory
|
||||
/usr/bin/stat: cannot statx '/home/zzz-no-such-r95': No such file or directory
|
||||
|
||||
=== ls -a /home/.zfs/snapshot ===
|
||||
/usr/bin/ls: cannot access '/home/.zfs/snapshot': No such file or directory
|
||||
=== ls -a /home ===
|
||||
.
|
||||
..
|
||||
.ssh
|
||||
felhom-repo
|
||||
=== tree /home/.zfs ===
|
||||
/home/.zfs [error opening dir]
|
||||
|
||||
0 directories, 0 files
|
||||
+165
@@ -0,0 +1,165 @@
|
||||
warm-up / positive control: /home
|
||||
........................ day 2026-09-01 done, batches-with-hits=0
|
||||
........................ day 2026-08-31 done, batches-with-hits=0
|
||||
### CONTROL RUN — same code path, /home injected into every batch; MUST hit 144/144
|
||||
HIT(2026-09-01 00:0x): /home
|
||||
HIT(2026-09-01 00:1x): /home
|
||||
HIT(2026-09-01 00:2x): /home
|
||||
HIT(2026-09-01 00:3x): /home
|
||||
HIT(2026-09-01 00:4x): /home
|
||||
HIT(2026-09-01 00:5x): /home
|
||||
.HIT(2026-09-01 01:0x): /home
|
||||
HIT(2026-09-01 01:1x): /home
|
||||
HIT(2026-09-01 01:2x): /home
|
||||
HIT(2026-09-01 01:3x): /home
|
||||
HIT(2026-09-01 01:4x): /home
|
||||
HIT(2026-09-01 01:5x): /home
|
||||
.HIT(2026-09-01 02:0x): /home
|
||||
HIT(2026-09-01 02:1x): /home
|
||||
HIT(2026-09-01 02:2x): /home
|
||||
HIT(2026-09-01 02:3x): /home
|
||||
HIT(2026-09-01 02:4x): /home
|
||||
HIT(2026-09-01 02:5x): /home
|
||||
.HIT(2026-09-01 03:0x): /home
|
||||
HIT(2026-09-01 03:1x): /home
|
||||
HIT(2026-09-01 03:2x): /home
|
||||
HIT(2026-09-01 03:3x): /home
|
||||
HIT(2026-09-01 03:4x): /home
|
||||
HIT(2026-09-01 03:5x): /home
|
||||
.HIT(2026-09-01 04:0x): /home
|
||||
HIT(2026-09-01 04:1x): /home
|
||||
HIT(2026-09-01 04:2x): /home
|
||||
HIT(2026-09-01 04:3x): /home
|
||||
HIT(2026-09-01 04:4x): /home
|
||||
HIT(2026-09-01 04:5x): /home
|
||||
.HIT(2026-09-01 05:0x): /home
|
||||
HIT(2026-09-01 05:1x): /home
|
||||
HIT(2026-09-01 05:2x): /home
|
||||
HIT(2026-09-01 05:3x): /home
|
||||
HIT(2026-09-01 05:4x): /home
|
||||
HIT(2026-09-01 05:5x): /home
|
||||
.HIT(2026-09-01 06:0x): /home
|
||||
HIT(2026-09-01 06:1x): /home
|
||||
HIT(2026-09-01 06:2x): /home
|
||||
HIT(2026-09-01 06:3x): /home
|
||||
HIT(2026-09-01 06:4x): /home
|
||||
HIT(2026-09-01 06:5x): /home
|
||||
.HIT(2026-09-01 07:0x): /home
|
||||
HIT(2026-09-01 07:1x): /home
|
||||
HIT(2026-09-01 07:2x): /home
|
||||
HIT(2026-09-01 07:3x): /home
|
||||
HIT(2026-09-01 07:4x): /home
|
||||
HIT(2026-09-01 07:5x): /home
|
||||
.HIT(2026-09-01 08:0x): /home
|
||||
HIT(2026-09-01 08:1x): /home
|
||||
HIT(2026-09-01 08:2x): /home
|
||||
HIT(2026-09-01 08:3x): /home
|
||||
HIT(2026-09-01 08:4x): /home
|
||||
HIT(2026-09-01 08:5x): /home
|
||||
.HIT(2026-09-01 09:0x): /home
|
||||
HIT(2026-09-01 09:1x): /home
|
||||
HIT(2026-09-01 09:2x): /home
|
||||
HIT(2026-09-01 09:3x): /home
|
||||
HIT(2026-09-01 09:4x): /home
|
||||
HIT(2026-09-01 09:5x): /home
|
||||
.HIT(2026-09-01 10:0x): /home
|
||||
HIT(2026-09-01 10:1x): /home
|
||||
HIT(2026-09-01 10:2x): /home
|
||||
HIT(2026-09-01 10:3x): /home
|
||||
HIT(2026-09-01 10:4x): /home
|
||||
HIT(2026-09-01 10:5x): /home
|
||||
.HIT(2026-09-01 11:0x): /home
|
||||
HIT(2026-09-01 11:1x): /home
|
||||
HIT(2026-09-01 11:2x): /home
|
||||
HIT(2026-09-01 11:3x): /home
|
||||
HIT(2026-09-01 11:4x): /home
|
||||
HIT(2026-09-01 11:5x): /home
|
||||
.HIT(2026-09-01 12:0x): /home
|
||||
HIT(2026-09-01 12:1x): /home
|
||||
HIT(2026-09-01 12:2x): /home
|
||||
HIT(2026-09-01 12:3x): /home
|
||||
HIT(2026-09-01 12:4x): /home
|
||||
HIT(2026-09-01 12:5x): /home
|
||||
.HIT(2026-09-01 13:0x): /home
|
||||
HIT(2026-09-01 13:1x): /home
|
||||
HIT(2026-09-01 13:2x): /home
|
||||
HIT(2026-09-01 13:3x): /home
|
||||
HIT(2026-09-01 13:4x): /home
|
||||
HIT(2026-09-01 13:5x): /home
|
||||
.HIT(2026-09-01 14:0x): /home
|
||||
HIT(2026-09-01 14:1x): /home
|
||||
HIT(2026-09-01 14:2x): /home
|
||||
HIT(2026-09-01 14:3x): /home
|
||||
HIT(2026-09-01 14:4x): /home
|
||||
HIT(2026-09-01 14:5x): /home
|
||||
.HIT(2026-09-01 15:0x): /home
|
||||
HIT(2026-09-01 15:1x): /home
|
||||
HIT(2026-09-01 15:2x): /home
|
||||
HIT(2026-09-01 15:3x): /home
|
||||
HIT(2026-09-01 15:4x): /home
|
||||
HIT(2026-09-01 15:5x): /home
|
||||
.HIT(2026-09-01 16:0x): /home
|
||||
HIT(2026-09-01 16:1x): /home
|
||||
HIT(2026-09-01 16:2x): /home
|
||||
HIT(2026-09-01 16:3x): /home
|
||||
HIT(2026-09-01 16:4x): /home
|
||||
HIT(2026-09-01 16:5x): /home
|
||||
.HIT(2026-09-01 17:0x): /home
|
||||
HIT(2026-09-01 17:1x): /home
|
||||
HIT(2026-09-01 17:2x): /home
|
||||
HIT(2026-09-01 17:3x): /home
|
||||
HIT(2026-09-01 17:4x): /home
|
||||
HIT(2026-09-01 17:5x): /home
|
||||
.HIT(2026-09-01 18:0x): /home
|
||||
HIT(2026-09-01 18:1x): /home
|
||||
HIT(2026-09-01 18:2x): /home
|
||||
HIT(2026-09-01 18:3x): /home
|
||||
HIT(2026-09-01 18:4x): /home
|
||||
HIT(2026-09-01 18:5x): /home
|
||||
.HIT(2026-09-01 19:0x): /home
|
||||
HIT(2026-09-01 19:1x): /home
|
||||
HIT(2026-09-01 19:2x): /home
|
||||
HIT(2026-09-01 19:3x): /home
|
||||
HIT(2026-09-01 19:4x): /home
|
||||
HIT(2026-09-01 19:5x): /home
|
||||
.HIT(2026-09-01 20:0x): /home
|
||||
HIT(2026-09-01 20:1x): /home
|
||||
HIT(2026-09-01 20:2x): /home
|
||||
HIT(2026-09-01 20:3x): /home
|
||||
HIT(2026-09-01 20:4x): /home
|
||||
HIT(2026-09-01 20:5x): /home
|
||||
.HIT(2026-09-01 21:0x): /home
|
||||
HIT(2026-09-01 21:1x): /home
|
||||
HIT(2026-09-01 21:2x): /home
|
||||
HIT(2026-09-01 21:3x): /home
|
||||
HIT(2026-09-01 21:4x): /home
|
||||
HIT(2026-09-01 21:5x): /home
|
||||
.HIT(2026-09-01 22:0x): /home
|
||||
HIT(2026-09-01 22:1x): /home
|
||||
HIT(2026-09-01 22:2x): /home
|
||||
HIT(2026-09-01 22:3x): /home
|
||||
HIT(2026-09-01 22:4x): /home
|
||||
HIT(2026-09-01 22:5x): /home
|
||||
.HIT(2026-09-01 23:0x): /home
|
||||
HIT(2026-09-01 23:1x): /home
|
||||
HIT(2026-09-01 23:2x): /home
|
||||
HIT(2026-09-01 23:3x): /home
|
||||
HIT(2026-09-01 23:4x): /home
|
||||
HIT(2026-09-01 23:5x): /home
|
||||
. day 2026-09-01 done, batches-with-hits=144
|
||||
|
||||
### REMAINING FIVE DAYS — no control injected
|
||||
........................ day 2026-08-30 done, batches-with-hits=0
|
||||
........................ day 2026-08-29 done, batches-with-hits=0
|
||||
........................ day 2026-08-28 done, batches-with-hits=0
|
||||
........................ day 2026-08-27 done, batches-with-hits=0
|
||||
........................ day 2026-08-26 done, batches-with-hits=0
|
||||
.......................### REMAINING DAYS, full 24h each, second granularity
|
||||
day 2026-08-30: hits=0
|
||||
day 2026-08-29: hits=0
|
||||
day 2026-08-28: hits=0
|
||||
day 2026-08-27: hits=0
|
||||
day 2026-08-26: hits=0
|
||||
day 2026-08-25: hits=0
|
||||
day 2026-08-24: hits=0
|
||||
### done
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
### CONTROL: exactly the sweep's batch shape (600 fake names) + 1 real path appended.
|
||||
### If the sweep can see a directory that exists, this MUST print /home. 6 batches tried.
|
||||
control batches: 6, batches that correctly returned exactly /home: 6
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
candidates: 126
|
||||
=== HITS ===
|
||||
/.zfs/shares/.
|
||||
/home
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
=== is rclone in the controller container? ===
|
||||
NO rclone in container
|
||||
=== does restic 0.14.0 know the rclone backend? (control: banana:) ===
|
||||
Fatal: parsing repository location failed: invalid backend
|
||||
If the repository is in a local directory, you need to add a `local:` prefix
|
||||
--- rclone: backend ---
|
||||
Fatal: unable to open repository at rclone:nosuchremote:/x: exec: "rclone": executable file not found in $PATH
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# phase2-the-attack — NOT RUN
|
||||
|
||||
The drill stopped at the end of Phase 1 on the operator's ruling. **This directory is empty on
|
||||
purpose, not by omission.**
|
||||
|
||||
Phase 1 established that no Storage Box snapshot is reachable from the box by any name (R-433), and
|
||||
that every other route to one is fenced (the Hetzner API, §11-D) or unavailable (a browser and a
|
||||
main-account credential nobody here holds). With no recovery leg, the deletion in Phase 2 would have
|
||||
destroyed real off-site history to buy only an alarm test that could not have fired at the size the
|
||||
runbook specifies — the detector needs a fall of more than half of 69, and one app's tag is ~9
|
||||
(R-435). Put to the operator as a two-option decision; the ruling was stop and report.
|
||||
|
||||
See `../README.md` and `felhom.eu/REPORT.md`.
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# phase3-the-alarm — NOT RUN
|
||||
|
||||
The drill stopped at the end of Phase 1 on the operator's ruling. **This directory is empty on
|
||||
purpose, not by omission.**
|
||||
|
||||
Phase 1 established that no Storage Box snapshot is reachable from the box by any name (R-433), and
|
||||
that every other route to one is fenced (the Hetzner API, §11-D) or unavailable (a browser and a
|
||||
main-account credential nobody here holds). With no recovery leg, the deletion in Phase 2 would have
|
||||
destroyed real off-site history to buy only an alarm test that could not have fired at the size the
|
||||
runbook specifies — the detector needs a fall of more than half of 69, and one app's tag is ~9
|
||||
(R-435). Put to the operator as a two-option decision; the ruling was stop and report.
|
||||
|
||||
See `../README.md` and `felhom.eu/REPORT.md`.
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# phase4-the-recovery — NOT RUN
|
||||
|
||||
The drill stopped at the end of Phase 1 on the operator's ruling. **This directory is empty on
|
||||
purpose, not by omission.**
|
||||
|
||||
Phase 1 established that no Storage Box snapshot is reachable from the box by any name (R-433), and
|
||||
that every other route to one is fenced (the Hetzner API, §11-D) or unavailable (a browser and a
|
||||
main-account credential nobody here holds). With no recovery leg, the deletion in Phase 2 would have
|
||||
destroyed real off-site history to buy only an alarm test that could not have fired at the size the
|
||||
runbook specifies — the detector needs a fall of more than half of 69, and one app's tag is ~9
|
||||
(R-435). Put to the operator as a two-option decision; the ruling was stop and report.
|
||||
|
||||
See `../README.md` and `felhom.eu/REPORT.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
# phase5-rearm — NOT RUN
|
||||
|
||||
The drill stopped at the end of Phase 1 on the operator's ruling. **This directory is empty on
|
||||
purpose, not by omission.**
|
||||
|
||||
Phase 1 established that no Storage Box snapshot is reachable from the box by any name (R-433), and
|
||||
that every other route to one is fenced (the Hetzner API, §11-D) or unavailable (a browser and a
|
||||
main-account credential nobody here holds). With no recovery leg, the deletion in Phase 2 would have
|
||||
destroyed real off-site history to buy only an alarm test that could not have fired at the size the
|
||||
runbook specifies — the detector needs a fall of more than half of 69, and one app's tag is ~9
|
||||
(R-435). Put to the operator as a two-option decision; the ruling was stop and report.
|
||||
|
||||
See `../README.md` and `felhom.eu/REPORT.md`.
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
=== A. off-site inventory UNCHANGED? (must still be 69) ===
|
||||
bookstack
|
||||
-----------------------------------------------------
|
||||
69 snapshots
|
||||
|
||||
=== B. account home contents (must be exactly .ssh + felhom-repo, nothing of mine) ===
|
||||
.
|
||||
..
|
||||
.ssh
|
||||
felhom-repo
|
||||
|
||||
=== C. repo top level (untouched) ===
|
||||
config
|
||||
data
|
||||
index
|
||||
keys
|
||||
locks
|
||||
snapshots
|
||||
|
||||
=== D. LAYER 1 CLEAN: ssh control sockets inside the container ===
|
||||
/tmp/r95-cm4-u629488-sub3
|
||||
removed
|
||||
verified: no control sockets left in container
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
=== LAYER 2 (guest 9201) + LAYER 3 (demo-hp PVE host) cleanup ===
|
||||
-- guest scratch before:
|
||||
.
|
||||
..
|
||||
cands.txt
|
||||
cmd.sh
|
||||
env.sh
|
||||
probe.txt
|
||||
sweep1.txt
|
||||
-- guest scratch after:
|
||||
ls: cannot access '/mnt/r95drill': No such file or directory
|
||||
-- guest /mnt now:
|
||||
.
|
||||
..
|
||||
felhom-drives
|
||||
sys_drive
|
||||
-- host /tmp/r95cmd.sh after:
|
||||
ls: cannot access '/tmp/r95cmd.sh': No such file or directory
|
||||
-- controller container healthy:
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.232.0 Up 6 hours (healthy)
|
||||
-- app containers healthy count:
|
||||
19
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
=== LAYER 4 (DooPlex) — shred the hub DB copy (it holds every host's secret) ===
|
||||
-rw-rw-r-- 1 kisfenyo kisfenyo 236482560 Sep 1 16:45 /tmp/claude-1000/hubdb.db
|
||||
ls: cannot access '/tmp/claude-1000/hubdb.db*': No such file or directory
|
||||
verified: hub DB copies gone
|
||||
|
||||
=== demo-felhom NOT TOUCHED — positive check, its own tier still reporting ok ===
|
||||
demo-hp: report 22221 @2026-09-01 14:45:18 snapshot_count=69 stats_known=True last_status=ok last_success=2026-09-01T02:17:57Z
|
||||
demo-felhom: report 22222 @2026-09-01 14:47:19 snapshot_count=10 stats_known=True last_status=ok last_success=2026-09-01T08:32:13Z
|
||||
|
||||
--- any offsite_snapshots_dropped events raised today? (must be only the constructed one) ---
|
||||
(3368, 'demo-hp', '2026-09-01 12:29:06', 'offsite_snapshots_dropped')
|
||||
second copy shredded
|
||||
Reference in New Issue
Block a user