BIGNIGHT teardown complete: VM destroyed, host deleted, ep0 peer gone, customer and its ep0 data kept
gates / gates (push) Successful in 19s
gates / gates (push) Successful in 19s
This commit is contained in:
@@ -40,7 +40,7 @@ ISO 1.27.1 sha `25637007…` found in the build output, not rebuilt. Venue: VM 3
|
|||||||
| 4 routines | Tier 1 ✓ · Tier 2 ✓ · whole-system local ✓ but apps down 8 min and a false PBS claim · guarded Update on a real bump ✓ 11 s, data intact · catalog reverted |
|
| 4 routines | Tier 1 ✓ · Tier 2 ✓ · whole-system local ✓ but apps down 8 min and a false PBS claim · guarded Update on a real bump ✓ 11 s, data intact · catalog reverted |
|
||||||
| 5 faults | F1 F2 F3 power cuts heal ≈ 4 min · F4 F5 drive pull/return honest, heals 91 s · F6 drive lost in backup: skipped apps reported success, alarm mails silenced · F7 disk 95 % holds, English banner, operator not told · F8 internet gone: LAN works, tunnel self-heals 9 s · **F9 controller killed: dead 33 min, nobody told — STOP** |
|
| 5 faults | F1 F2 F3 power cuts heal ≈ 4 min · F4 F5 drive pull/return honest, heals 91 s · F6 drive lost in backup: skipped apps reported success, alarm mails silenced · F7 disk 95 % holds, English banner, operator not told · F8 internet gone: LAN works, tunnel self-heals 9 s · **F9 controller killed: dead 33 min, nobody told — STOP** |
|
||||||
| 6 morning after | apps healthy · 1 false label (downgrade offered as update) · local BookStack restore ✓ 24 s |
|
| 6 morning after | apps healthy · 1 false label (downgrade offered as update) · local BookStack restore ✓ 24 s |
|
||||||
| 7 teardown | layer 1 done (VM, disks, ISO, harness files); layer 2 host delete + ep0 peer: see the audit's teardown section; customer `tester-1` **kept**; its ep0 data **kept, stated** |
|
| 7 teardown | machine: VM 333 + disks, ISO, harness files removed, storage back to pre-drill levels · host: `tester-1-a61396` deleted, ep0 peer gone · hub: customer `tester-1` **kept**; its ep0 data (1 snapshot dir) **kept, stated** · 9201/9202 untouched · secrets shredded |
|
||||||
|
|
||||||
## 4. Rows (register 221 → 237)
|
## 4. Rows (register 221 → 237)
|
||||||
|
|
||||||
|
|||||||
@@ -93,8 +93,9 @@ truth table** is `evidence-bignight-2026-09-14/alarm-truth-table.md`.
|
|||||||
firewall table left. `pvesm status`: `nvme-scratch` back to 10 140 556 KiB (10 134 820 before the drill), `local`
|
firewall table left. `pvesm status`: `nvme-scratch` back to 10 140 556 KiB (10 134 820 before the drill), `local`
|
||||||
23 071 188 KiB (23 041 736 before), `local-lvm` 44.17 % unchanged all night. Evidence `teardown-before.txt`,
|
23 071 188 KiB (23 041 736 before), `local-lvm` 44.17 % unchanged all night. Evidence `teardown-before.txt`,
|
||||||
`teardown-layer1-machine.txt`. `pct fstrim` not applied: only deleted qcow2 files on a dir storage were used.
|
`teardown-layer1-machine.txt`. `pct fstrim` not applied: only deleted qcow2 files on a dir storage were used.
|
||||||
- **Host (hub record + ep0 peer):** see `teardown-layer3-hub.txt` — the host record `tester-1-a61396` is deleted once
|
- **Host (hub record + ep0 peer):** host `tester-1-a61396` deleted at 22:59:21Z once stale (the first attempt was refused
|
||||||
the hub marks it stale, and its WireGuard peer on ep0 is checked gone read-only.
|
for a missing `confirm_host_id` — harness); host page 404; after the 23:04:13Z peer sync ep0 lists **no peer
|
||||||
|
`10.77.0.5`** (read-only check). Evidence `teardown-layer3-hub.txt`.
|
||||||
- **Hub customer:** **„Tester 1" is KEPT** (the volunteer's record). Its off-site data on ep0 — namespace `tester-1`,
|
- **Hub customer:** **„Tester 1" is KEPT** (the volunteer's record). Its off-site data on ep0 — namespace `tester-1`,
|
||||||
one snapshot directory from the doorstep walk — is **kept, stated**, for the operator to rule (R-511 context).
|
one snapshot directory from the doorstep walk — is **kept, stated**, for the operator to rule (R-511 context).
|
||||||
- **Untouched, and checked:** demo-hp 9201 and 9202 (running throughout; only read-only loopback probes, R-513);
|
- **Untouched, and checked:** demo-hp 9201 and 9202 (running throughout; only read-only loopback probes, R-513);
|
||||||
|
|||||||
@@ -693,3 +693,18 @@ touched and are not trimmed.
|
|||||||
**ep0 read-only before the host delete** (`teardown-ep0-before-host-delete.txt`, 22:26:02Z): WireGuard peer
|
**ep0 read-only before the host delete** (`teardown-ep0-before-host-delete.txt`, 22:26:02Z): WireGuard peer
|
||||||
`10.77.0.5/32` present; namespaces `demo-felhom demo-hp tester-1`; `tester-1/ct/9201` with 1 snapshot directory (the
|
`10.77.0.5/32` present; namespaces `demo-felhom demo-hp tester-1`; `tester-1/ct/9201` with 1 snapshot directory (the
|
||||||
doorstep walk's data). Nothing was written to ep0 tonight.
|
doorstep walk's data). Nothing was written to ep0 tonight.
|
||||||
|
|
||||||
|
**Layer 3 — host record and ep0 peer** (`teardown-layer3-hub.txt`): the destroyed box's last host report reached the
|
||||||
|
hub at 22:23:45Z; delete-impact turned `stale · deletable` at ≈ 22:54Z. **Harness slip:** the first `POST
|
||||||
|
/hosts/tester-1-a61396/delete` (22:54:40Z) sent no fields → **400**, hub `host delete refused: … confirm mismatch`; the
|
||||||
|
host record and the ep0 peer stayed (ep0 read at 22:58:42Z: peer 10.77.0.5 present, tester-1 snapshot dirs 1). The
|
||||||
|
handler wants `confirm_host_id` = the host id (its test D3). Attempt 2 below.
|
||||||
|
**Attempt 2 (22:59:21Z)** with `confirm_host_id=tester-1-a61396` → **303 `/hosts`**; hub `host deleted: tester-1-a61396
|
||||||
|
(escrow deleted: false)`; host page **404**; not on `/hosts`; **customer `tester-1` page 200 — kept**. The first ep0
|
||||||
|
re-read (23:00:23Z) still showed the peer, because the last peer sync (22:59:13Z) predates the delete by 8 s — harness
|
||||||
|
read too early; re-read after the next sync below.
|
||||||
|
**ep0 after the next sync** (23:04:13Z `wgsync: pushed 4 peers`; read-only 23:04:33Z): **peer `10.77.0.5` gone (0)**,
|
||||||
|
control peer `10.77.0.3` present; namespaces `demo-felhom demo-hp tester-1`; `tester-1` still holds **1 snapshot
|
||||||
|
directory — kept, stated** for the operator. **Teardown complete: machine, host, hub.** The night's secrets in the
|
||||||
|
session scratchpad (hub password copy, break-glass password, passphrase, codes, app passwords, bind token) shredded at
|
||||||
|
23:04Z; a search of the scratchpad for leftover tokens found none.
|
||||||
|
|||||||
@@ -0,0 +1,5 @@
|
|||||||
|
=== ep0 read-only 2026-09-14T22:26:02Z
|
||||||
|
peer 10.77.0.5 present: 1
|
||||||
|
demo-felhom demo-hp tester-1
|
||||||
|
tester-1 ct: 9201
|
||||||
|
tester-1 snapshot dirs: 1
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
+ qm stop 333 --timeout 30
|
||||||
|
+ qm destroy 333 --purge 1 --destroy-unreferenced-disks 1
|
||||||
|
purging VM 333 from related configurations..
|
||||||
|
+ rm -f /var/lib/vz/template/iso/felhom-installer-1.27.1-pve9.2-1.iso
|
||||||
|
+ rm -rf /root/bn
|
||||||
|
+ set +x
|
||||||
|
=== AFTER layer 1 2026-09-14T22:25:58Z
|
||||||
|
VMID Status Lock Name
|
||||||
|
9201 running demo-hp
|
||||||
|
9202 running demo-hp-scratch
|
||||||
|
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
|
||||||
|
felhom-pbs pbs active 0 0 0 0.00%
|
||||||
|
local dir active 40453376 23071188 15295072 57.03%
|
||||||
|
local-lvm lvmthin active 56487936 24950721 31537214 44.17%
|
||||||
|
nvme-scratch dir active 983379700 10140556 923212532 1.03%
|
||||||
|
/dev/nvme0n1 1006980812800 10383929344 945369632768 2% /mnt/hdd_1
|
||||||
|
/mnt/hdd_1/images/:
|
||||||
|
total 12
|
||||||
|
drwxr-xr-x 3 root root 4096 Sep 15 00:25 .
|
||||||
|
drwxr-xr-x 9 root root 4096 Sep 13 22:16 ..
|
||||||
|
drwxr----- 2 root root 4096 Sep 13 22:18 9202
|
||||||
|
|
||||||
|
/var/lib/vz/template/iso/:
|
||||||
|
total 8
|
||||||
|
drwxr-xr-x 2 root root 4096 Sep 15 00:25 .
|
||||||
|
drwxr-xr-x 4 root root 4096 Aug 21 17:42 ..
|
||||||
|
ls: cannot access '/root/bn': No such file or directory
|
||||||
|
table inet felhom_oob
|
||||||
|
0
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
=== LAYER 3 start 2026-09-14T22:26:33Z
|
||||||
|
22:26:33 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:27:33 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:28:33 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:29:34 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:30:34 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:31:34 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:32:35 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:33:35 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:34:35 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:35:35 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:36:36 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:37:36 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:38:36 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:39:37 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:40:37 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:41:37 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:42:37 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:43:38 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:44:38 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:45:38 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:46:38 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:47:38 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:48:39 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:49:39 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:50:39 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:51:40 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:52:40 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:53:40 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true}
|
||||||
|
22:54:40 impact: {"deletable":true,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"stale","wg_peer_bound":true}
|
||||||
|
=== host delete 2026-09-14T22:54:40Z
|
||||||
|
HTTP/1.1 400 Bad Request
|
||||||
|
host page: 200
|
||||||
|
customer page tester-1 (must stay): 200
|
||||||
|
tester-1-a61396 on /hosts: 2
|
||||||
|
2026/09/15 00:54:41 [WARN] host delete refused: tester-1-a61396 confirm mismatch
|
||||||
|
2026/09/15 00:54:13 [INFO] wgsync: pushed 5 peers to 167.233.158.164:22
|
||||||
|
=== ep0 read-only 2026-09-14T22:58:42Z
|
||||||
|
peer 10.77.0.5: 1 control 10.77.0.3: 1
|
||||||
|
tester-1 snapshot dirs: 1
|
||||||
|
=== host delete, attempt 2 (with confirm_host_id) 2026-09-14T22:59:21Z
|
||||||
|
impact: {"deletable":true,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"stale","wg_peer_bound":true}
|
||||||
|
HTTP/1.1 303 See Other
|
||||||
|
Location: /hosts
|
||||||
|
host page: 404
|
||||||
|
customer page tester-1 (must stay): 200
|
||||||
|
tester-1-a61396 on /hosts: 0
|
||||||
|
2026/09/15 00:59:21 [INFO] host deleted: tester-1-a61396 (escrow deleted: false)
|
||||||
|
2026/09/15 00:54:13 [INFO] wgsync: pushed 5 peers to 167.233.158.164:22
|
||||||
|
2026/09/15 00:59:13 [INFO] wgsync: pushed 5 peers to 167.233.158.164:22
|
||||||
|
=== ep0 read-only 2026-09-14T23:00:23Z
|
||||||
|
peer 10.77.0.5: 1 control 10.77.0.3: 1
|
||||||
|
tester-1 snapshot dirs: 1
|
||||||
|
=== peer sync after the delete
|
||||||
|
2026/09/15 00:59:13 [INFO] wgsync: pushed 5 peers to 167.233.158.164:22
|
||||||
|
2026/09/15 01:04:13 [INFO] wgsync: pushed 4 peers to 167.233.158.164:22
|
||||||
|
=== ep0 read-only 2026-09-14T23:04:33Z
|
||||||
|
peer 10.77.0.5: 0 control 10.77.0.3: 1
|
||||||
|
namespaces: demo-felhom demo-hp tester-1
|
||||||
|
tester-1 snapshot dirs: 1
|
||||||
Reference in New Issue
Block a user