felhom.eu/documentation/audits/evidence-drill-r95-recovery-2026-09-01

Evidence — DRILL: the deletion we said is survivable (R-95 recovery drill, 2026-09-01)

The drill STOPPED at the end of Phase 1, on the operator's ruling, before any destructive step. No delete verb was issued against any live store. No byte on either Storage Box sub-account was written, moved or removed. Every probe in this directory is a read: ls, stat, tree, du, df, rsync --list-only, restic snapshots.

Why it stopped

Phase 1 was supposed to find a Storage Box snapshot so Phase 4 could recover from it. It found that no snapshot is reachable from the box by any name, and that no unfenced route to one exists for CC at all. The recovery leg therefore could not run, and deleting a real app's off-site history would have bought only an alarm test that — measured against the shipped threshold — could not have fired at the size the runbook specifies. Put to the operator as a two-option decision; the ruling was stop and report.

Files

file	what it is
phase0-ground-truth/01-offsite-inventory.txt	demo-hp's full off-site inventory — 69 restic snapshots, 9 apps, ids + tags + paths. The baseline.
phase0-ground-truth/02-hub-view.txt	the hub's latest offsite object for demo-hp (snapshot_count: 69, stats_known: true, last_status: ok). Cross-checks the restic count exactly.
phase1-find-the-snapshot/01-probe-tree.txt	SFTP probes of /.zfs, /.zfs/snapshot, /home/.zfs, with a positive and a negative control. Reproduces R-432.
phase1-find-the-snapshot/02-shell-and-rsync-doors.txt	two independent tools (a real shell on port 23, and rsync --list-only) agree the tree lists empty. Both controlled.
phase1-find-the-snapshot/03-shell-capabilities.txt	the port-23 restricted shell's full help — the command set the credential actually reaches.
phase1-find-the-snapshot/04-enumeration-attempts.txt	df/stat/tree/du on the snapshot door. Where the st_dev split was found.
phase1-find-the-snapshot/05-home-zfs-door.txt	/home/.zfs does not exist — three tools, negative control in the same run.
phase1-find-the-snapshot/06-name-sweep.txt	the name sweep: nine full days, second granularity, 777,600 candidate names, zero hits.
phase1-find-the-snapshot/07-sweep-control.txt	the control for that sweep — the identical 600-name batch shape with one real path appended; 6/6 batches returned it. Without this the zero result would prove nothing.
phase1-find-the-snapshot/08-alt-formats.txt	126 non-timestamp name shapes and alternative snapshot paths. Only the controls resolved.
phase1-find-the-snapshot/09-rclone-restic-lead.txt	the rclone: backend measurement behind R-436, with the banana: invalid-backend control.
phase6-teardown/01-verify-then-clean.txt	the store is untouched: still 69 snapshots, home is exactly .ssh + felhom-repo, repo top level intact.
phase6-teardown/02-guest-and-host-clean.txt	scratch removed from the guest and the PVE host; controller 0.232.0 healthy, 19 healthy containers.
phase6-teardown/03-dooplex-clean-and-final-state.txt	hub DB copies shredded (both, -wal included); both customers' tiers still ok; no alarm event raised by this session.

The method that makes the negative result trustworthy

A sweep that finds nothing is worthless unless it is shown it could have found something. The oracle is a batched stat -c %n over the port-23 shell: 500–600 paths per round trip, and stdout carries only the paths that exist. 07-sweep-control.txt runs the identical batch shape with /home appended and gets /home back, 6 times out of 6. So the zero in 06-name-sweep.txt is a measurement, not a silence.

Not done, and why

• No Hetzner API call. Fenced by the runbook (§11-D). The hub's own API client has no snapshot method at all, so even unfenced it would have needed new code.
• No panel action. No browser on DooPlex, and "Restore snapshot" is fenced in any case — it rolls back the whole Storage Box and deletes newer snapshots.
• demo-felhom never addressed. Every storage-box connection in this session authenticated as u629488-sub3, demo-hp's own sub-account. Its tier is verified still reporting ok in phase6-teardown/03-*.