diff --git a/REPORT-bignight-2026-09-14.md b/REPORT-bignight-2026-09-14.md index 428baeec..820f3dfa 100644 --- a/REPORT-bignight-2026-09-14.md +++ b/REPORT-bignight-2026-09-14.md @@ -40,7 +40,7 @@ ISO 1.27.1 sha `25637007…` found in the build output, not rebuilt. Venue: VM 3 | 4 routines | Tier 1 ✓ · Tier 2 ✓ · whole-system local ✓ but apps down 8 min and a false PBS claim · guarded Update on a real bump ✓ 11 s, data intact · catalog reverted | | 5 faults | F1 F2 F3 power cuts heal ≈ 4 min · F4 F5 drive pull/return honest, heals 91 s · F6 drive lost in backup: skipped apps reported success, alarm mails silenced · F7 disk 95 % holds, English banner, operator not told · F8 internet gone: LAN works, tunnel self-heals 9 s · **F9 controller killed: dead 33 min, nobody told — STOP** | | 6 morning after | apps healthy · 1 false label (downgrade offered as update) · local BookStack restore ✓ 24 s | -| 7 teardown | layer 1 done (VM, disks, ISO, harness files); layer 2 host delete + ep0 peer: see the audit's teardown section; customer `tester-1` **kept**; its ep0 data **kept, stated** | +| 7 teardown | machine: VM 333 + disks, ISO, harness files removed, storage back to pre-drill levels · host: `tester-1-a61396` deleted, ep0 peer gone · hub: customer `tester-1` **kept**; its ep0 data (1 snapshot dir) **kept, stated** · 9201/9202 untouched · secrets shredded | ## 4. Rows (register 221 → 237) diff --git a/documentation/audits/BIGNIGHT-household-month-2026-09-14.md b/documentation/audits/BIGNIGHT-household-month-2026-09-14.md index 7f6b898f..e7aa986e 100644 --- a/documentation/audits/BIGNIGHT-household-month-2026-09-14.md +++ b/documentation/audits/BIGNIGHT-household-month-2026-09-14.md @@ -93,8 +93,9 @@ truth table** is `evidence-bignight-2026-09-14/alarm-truth-table.md`. firewall table left. `pvesm status`: `nvme-scratch` back to 10 140 556 KiB (10 134 820 before the drill), `local` 23 071 188 KiB (23 041 736 before), `local-lvm` 44.17 % unchanged all night. Evidence `teardown-before.txt`, `teardown-layer1-machine.txt`. `pct fstrim` not applied: only deleted qcow2 files on a dir storage were used. -- **Host (hub record + ep0 peer):** see `teardown-layer3-hub.txt` — the host record `tester-1-a61396` is deleted once - the hub marks it stale, and its WireGuard peer on ep0 is checked gone read-only. +- **Host (hub record + ep0 peer):** host `tester-1-a61396` deleted at 22:59:21Z once stale (the first attempt was refused + for a missing `confirm_host_id` — harness); host page 404; after the 23:04:13Z peer sync ep0 lists **no peer + `10.77.0.5`** (read-only check). Evidence `teardown-layer3-hub.txt`. - **Hub customer:** **„Tester 1" is KEPT** (the volunteer's record). Its off-site data on ep0 — namespace `tester-1`, one snapshot directory from the doorstep walk — is **kept, stated**, for the operator to rule (R-511 context). - **Untouched, and checked:** demo-hp 9201 and 9202 (running throughout; only read-only loopback probes, R-513); diff --git a/documentation/audits/evidence-bignight-2026-09-14/journal.md b/documentation/audits/evidence-bignight-2026-09-14/journal.md index 62128613..2ca9df9a 100644 --- a/documentation/audits/evidence-bignight-2026-09-14/journal.md +++ b/documentation/audits/evidence-bignight-2026-09-14/journal.md @@ -693,3 +693,18 @@ touched and are not trimmed. **ep0 read-only before the host delete** (`teardown-ep0-before-host-delete.txt`, 22:26:02Z): WireGuard peer `10.77.0.5/32` present; namespaces `demo-felhom demo-hp tester-1`; `tester-1/ct/9201` with 1 snapshot directory (the doorstep walk's data). Nothing was written to ep0 tonight. + +**Layer 3 — host record and ep0 peer** (`teardown-layer3-hub.txt`): the destroyed box's last host report reached the +hub at 22:23:45Z; delete-impact turned `stale · deletable` at ≈ 22:54Z. **Harness slip:** the first `POST +/hosts/tester-1-a61396/delete` (22:54:40Z) sent no fields → **400**, hub `host delete refused: … confirm mismatch`; the +host record and the ep0 peer stayed (ep0 read at 22:58:42Z: peer 10.77.0.5 present, tester-1 snapshot dirs 1). The +handler wants `confirm_host_id` = the host id (its test D3). Attempt 2 below. +**Attempt 2 (22:59:21Z)** with `confirm_host_id=tester-1-a61396` → **303 `/hosts`**; hub `host deleted: tester-1-a61396 +(escrow deleted: false)`; host page **404**; not on `/hosts`; **customer `tester-1` page 200 — kept**. The first ep0 +re-read (23:00:23Z) still showed the peer, because the last peer sync (22:59:13Z) predates the delete by 8 s — harness +read too early; re-read after the next sync below. +**ep0 after the next sync** (23:04:13Z `wgsync: pushed 4 peers`; read-only 23:04:33Z): **peer `10.77.0.5` gone (0)**, +control peer `10.77.0.3` present; namespaces `demo-felhom demo-hp tester-1`; `tester-1` still holds **1 snapshot +directory — kept, stated** for the operator. **Teardown complete: machine, host, hub.** The night's secrets in the +session scratchpad (hub password copy, break-glass password, passphrase, codes, app passwords, bind token) shredded at +23:04Z; a search of the scratchpad for leftover tokens found none. diff --git a/documentation/audits/evidence-bignight-2026-09-14/teardown-ep0-before-host-delete.txt b/documentation/audits/evidence-bignight-2026-09-14/teardown-ep0-before-host-delete.txt new file mode 100644 index 00000000..4172958d --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/teardown-ep0-before-host-delete.txt @@ -0,0 +1,5 @@ +=== ep0 read-only 2026-09-14T22:26:02Z +peer 10.77.0.5 present: 1 +demo-felhom demo-hp tester-1 +tester-1 ct: 9201 +tester-1 snapshot dirs: 1 diff --git a/documentation/audits/evidence-bignight-2026-09-14/teardown-layer1-machine.txt b/documentation/audits/evidence-bignight-2026-09-14/teardown-layer1-machine.txt new file mode 100644 index 00000000..c65558fa --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/teardown-layer1-machine.txt @@ -0,0 +1,29 @@ ++ qm stop 333 --timeout 30 ++ qm destroy 333 --purge 1 --destroy-unreferenced-disks 1 +purging VM 333 from related configurations.. ++ rm -f /var/lib/vz/template/iso/felhom-installer-1.27.1-pve9.2-1.iso ++ rm -rf /root/bn ++ set +x +=== AFTER layer 1 2026-09-14T22:25:58Z +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +Name Type Status Total (KiB) Used (KiB) Available (KiB) % +felhom-pbs pbs active 0 0 0 0.00% +local dir active 40453376 23071188 15295072 57.03% +local-lvm lvmthin active 56487936 24950721 31537214 44.17% +nvme-scratch dir active 983379700 10140556 923212532 1.03% +/dev/nvme0n1 1006980812800 10383929344 945369632768 2% /mnt/hdd_1 +/mnt/hdd_1/images/: +total 12 +drwxr-xr-x 3 root root 4096 Sep 15 00:25 . +drwxr-xr-x 9 root root 4096 Sep 13 22:16 .. +drwxr----- 2 root root 4096 Sep 13 22:18 9202 + +/var/lib/vz/template/iso/: +total 8 +drwxr-xr-x 2 root root 4096 Sep 15 00:25 . +drwxr-xr-x 4 root root 4096 Aug 21 17:42 .. +ls: cannot access '/root/bn': No such file or directory +table inet felhom_oob +0 diff --git a/documentation/audits/evidence-bignight-2026-09-14/teardown-layer3-hub.txt b/documentation/audits/evidence-bignight-2026-09-14/teardown-layer3-hub.txt new file mode 100644 index 00000000..41d73081 --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/teardown-layer3-hub.txt @@ -0,0 +1,60 @@ +=== LAYER 3 start 2026-09-14T22:26:33Z +22:26:33 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:27:33 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:28:33 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:29:34 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:30:34 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:31:34 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:32:35 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:33:35 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:34:35 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:35:35 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:36:36 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:37:36 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:38:36 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:39:37 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:40:37 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:41:37 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:42:37 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:43:38 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:44:38 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:45:38 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:46:38 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:47:38 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:48:39 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:49:39 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:50:39 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:51:40 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:52:40 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:53:40 impact: {"deletable":false,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"ok","wg_peer_bound":true} +22:54:40 impact: {"deletable":true,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"stale","wg_peer_bound":true} +=== host delete 2026-09-14T22:54:40Z +HTTP/1.1 400 Bad Request +host page: 200 +customer page tester-1 (must stay): 200 +tester-1-a61396 on /hosts: 2 +2026/09/15 00:54:41 [WARN] host delete refused: tester-1-a61396 confirm mismatch +2026/09/15 00:54:13 [INFO] wgsync: pushed 5 peers to 167.233.158.164:22 +=== ep0 read-only 2026-09-14T22:58:42Z +peer 10.77.0.5: 1 control 10.77.0.3: 1 +tester-1 snapshot dirs: 1 +=== host delete, attempt 2 (with confirm_host_id) 2026-09-14T22:59:21Z +impact: {"deletable":true,"escrow_present":false,"guests":1,"log_bundles":0,"pbs_secret_present":false,"recovery_present":true,"reports":24,"status":"stale","wg_peer_bound":true} +HTTP/1.1 303 See Other +Location: /hosts +host page: 404 +customer page tester-1 (must stay): 200 +tester-1-a61396 on /hosts: 0 +2026/09/15 00:59:21 [INFO] host deleted: tester-1-a61396 (escrow deleted: false) +2026/09/15 00:54:13 [INFO] wgsync: pushed 5 peers to 167.233.158.164:22 +2026/09/15 00:59:13 [INFO] wgsync: pushed 5 peers to 167.233.158.164:22 +=== ep0 read-only 2026-09-14T23:00:23Z +peer 10.77.0.5: 1 control 10.77.0.3: 1 +tester-1 snapshot dirs: 1 +=== peer sync after the delete +2026/09/15 00:59:13 [INFO] wgsync: pushed 5 peers to 167.233.158.164:22 +2026/09/15 01:04:13 [INFO] wgsync: pushed 4 peers to 167.233.158.164:22 +=== ep0 read-only 2026-09-14T23:04:33Z +peer 10.77.0.5: 0 control 10.77.0.3: 1 +namespaces: demo-felhom demo-hp tester-1 +tester-1 snapshot dirs: 1