From 983d08275c7a64d1037e1990071439f91cbf971a Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 14 Sep 2026 23:12:48 +0200 Subject: [PATCH] BIGNIGHT: audit page draft (Phases 1-4, F1-F7); F8 in progress --- .../BIGNIGHT-household-month-2026-09-14.md | 73 +++++++++++++++++++ .../evidence-bignight-2026-09-14/journal.md | 22 ++++++ ...attempt1-HARNESS-SLIP-rule-not-applied.txt | 12 +++ .../phase5/F8-internet-gone.txt | 42 ++++++++--- .../F8/attempt1-poll-internet-NOT-cut.txt | 8 ++ .../phase5/F8/guest-wan-probe.txt | 2 + .../phase5/F8/hub-and-dashboard-poll.txt | 12 +++ 7 files changed, 160 insertions(+), 11 deletions(-) create mode 100644 documentation/audits/BIGNIGHT-household-month-2026-09-14.md create mode 100644 documentation/audits/evidence-bignight-2026-09-14/phase5/F8-attempt1-HARNESS-SLIP-rule-not-applied.txt create mode 100644 documentation/audits/evidence-bignight-2026-09-14/phase5/F8/attempt1-poll-internet-NOT-cut.txt create mode 100644 documentation/audits/evidence-bignight-2026-09-14/phase5/F8/guest-wan-probe.txt create mode 100644 documentation/audits/evidence-bignight-2026-09-14/phase5/F8/hub-and-dashboard-poll.txt diff --git a/documentation/audits/BIGNIGHT-household-month-2026-09-14.md b/documentation/audits/BIGNIGHT-household-month-2026-09-14.md new file mode 100644 index 00000000..138df241 --- /dev/null +++ b/documentation/audits/BIGNIGHT-household-month-2026-09-14.md @@ -0,0 +1,73 @@ +# BIGNIGHT — a household's first month, compressed into one night (2026-09-14/15) + +**Interventions a customer could not have made: Phase 2 = 2, Phase 3 = 0.** +**Ready for a volunteer: NO** — the dashboard link does not open through the tunnel (R-510), a box installed for an +existing customer gets no bind mail (R-509), and every box's file manager opens with `admin` / `admin` (R-513). + +Brief: `drills/BIGNIGHT-2026-09-14.md`. Evidence: `evidence-bignight-2026-09-14/` — `journal.md` (every observable in +order), `alarm-truth-table.md`, `screens/`, `box-logs-phase2..4/`, `phase3/`, `phase4/`, `phase5/F1..F12`, `phase6/`, +`teardown-*.txt`. Architecture read first: `00-capability-map.md`, `07-backup-architecture.md` §6 (the tiers), +`09-update-architecture.md` §3 (decisions 1–9). + +## Venue + +VM 333 on demo-hp: ISO **1.27.1** (sha `25637007…`, found in the build output, not rebuilt), q35/OVMF, 4 cores, +**16 GB** (the HP has 30 GB; 9201+9202 used ≈ 5.3 GB), system disk 200 G + data disk 100 G added after the install, +both qcow2 on `nvme-scratch` (`/mnt/hdd_1`, its root). Customer „Tester 1" (`tester-1`, `enkicsifelhom.hu`, +`tester1@felhom.eu`). Baselines: controller `406755fa` v0.242.0 · agent `4586f0f7` v0.130.0 · felhom.eu `a4d68441` +hub v0.113.0 · catalog `6d6eec30`. Harness substitutions as the earlier walks: U.S. keyboard (H2), auto-reboot +unticked and ISO detached (H3), text-mode installer (H4). + +**Off-site, as found:** the record has the DR tier (PBS on ep0) ticked and **restic off-site off**; ticking it +provisions a Hetzner Storage Box (money, fenced), so it was not ticked. The DR tier then could not provision on the new +box (R-511). **This box had no off-site tier of any kind; nothing was written to ep0.** + +## Phase 2 — the first hour (mail real this time) + +| step | result | +|---|---| +| install (one disk) | ≤ 2 m 45 s copying; the one-disk screen offers no choice; hostname and e-mail typed per the guide | +| first screen | Felhom Hungarian text only (no `:8006` line) ✓; pairing banner repeats 3× after bind (R-214 class) | +| data disk | hot-added; **nothing on dashboard, launcher or mail mentions it**; found under Tárhely → „Új meghajtó inicializálása"; enrolled in 2 s. A household would not know to enrol it | +| bind mail | **none in 10 min** for an existing customer → **R-509 (P1), I1**: operator pressed „Send self-bind link"; mail arrived in 1 s | +| self-bind link | **works end to end** (first walk to exercise it): code + Tulajdonosi jelmondat → „Sikeres összekötés." | +| setup-code mail | arrived 54 s after bind — the **reinstall** mail („újratelepült … A korábbi jelszavad már nem érvényes"), not the first-install mail the guide names; **the mailed code worked** | +| version | controller 0.242.0 + agent 0.130.0, current, no self-update needed ✓ | +| **gate: dashboard via tunnel** | **FAIL** — 502 ×9: route reaches the box but lacks „No TLS Verify" → **R-510 (P1), I2**; LAN address used from then on | + +## Phase 3 — twelve apps, seeded through their front doors + +All 12 deployed on the default box — **the memory guard never refused** (guest 11 828 MB; ≈ 3.5 GB used with 12 apps). +Every app „Fut · Naprakész" (12/12). Seeds: BookStack 5 Hungarian pages + 2 attachments (sha equal) + 2nd user + +delete/undo; Docmost space + 5 docs + rename/delete/restore; PrivateBin 5 encrypted pastes incl. 10-min expiry; +Gokapi 3 uploads incl. 50 MB (stranger download sha equal); Nextcloud 200 JPEG + 20 PDF, share with 2nd user, delete ++ trash restore (sha equal); Immich 200 photos, ML settled in ≈ 6 min; Vaultwarden 10 entries + attachment (client +crypto); **Paperless-ngx 20 PDFs → 0 documents (OOM, R-514)**; Jellyfin video via the product's SMB share → library +scan 5 s → stream; Mealie 5 recipes + meal plan; Uptime Kuma 3 monitors (its first screen is English, R-516; the +box's own names show DOWN through R-510); AdventureLog trip with visits (photos fail from a non-browser client — R-483 +class). Interventions: **0**. + +Findings: **R-512** Vaultwarden open signup with a read-only close control · **R-513 (P1, security)** FileBrowser +`admin/admin` on every box, demo-hp's login page public · **R-514** Paperless OOM silently · **R-515** Paperless card's +wrong login · **R-516** English strings enumerated. + +## Phase 4 — a month of routines + +Tier 1 run 2 m 08 s ✓ · Tier 2 12 apps in 17 s (drive apps state-only, as stated) ✓ · off-site: none to run or verify · +whole-system „Mentés most": local 8.9 GB in 362 s ✓, then the absent PBS tier failed **with every app stopped ≈ 7 m 45 s** +under „csak néhány másodpercre" (**R-518**) and the page afterwards claimed a current full backup and a remote copy +that do not exist (**R-517, P1**). Hub: box ok, 24/24 containers, drive shown, true `whole_guest_backup_failed` mailed. +Guarded Update on a real bump (privatebin 2.0.5 → 2.0.6, catalog `d5d91e0`, reverted `a161ccb` in the same phase): +reached the box 14 m 25 s after the push, „Frissítés elérhető — ma", **DONE in 11 s, data intact** ✓. + +## Phase 5 — the accidents (five measures each: customer saw · box did · time · alarm true? · alarm missed) + +| # | fault | what the customer saw | what the box did by itself | steady state | alarm fired / true? | should have fired, did not | +|---|---|---|---|---|---|---| +| F1 | power cut 61 s, family on 3 apps | all apps down ≈ 3½ min, re-login | all 12 back on same images; boot reconciler started paperless | **4 m 03 s** | `controller_started` / true | — | +| F2 | power cut during nightly backup (adventurelog stopped for its dump) | pages say „21:40 OK", no word of interruption | app-stop guard restarted adventurelog ✓; all 12 same images | 4 m 05 s | `backup_failed (error)` + mail / **true** | customer notice (R-519) | +| F3 | power cut in „pulling" of a guarded Update (nextcloud, same version) | „Fut · Naprakész", nothing about the update | back on same images, pin consistent; no journal trace | 3 m 50 s | `controller_started` / true | untestable same-version (R-520) | +| F4 | data drive unplugged under running apps | honest Hungarian: „Meghajtó leválasztva", „Hiányzó tárhely … Csatlakoztasd újra"; raw UTC time, banner ×2 | drive apps stopped by +38 s; **system-disk apps kept running** ✓; path failed closed (I/O error) | — | `storage_disconnected (error)` + 4 × `app_start_failed`, 5 mails / true, redundant | household mail (R-521) | +| F5 | drive out 30 min, then back | badge „Aktív" at once; stale banner cleared within 10 min | re-bound as `sdc` by UUID, ext4 recovery, restarted the 4 apps | **91 s** | `storage_reconnected (info)` / true | — | +| F6 | drive unplugged during a backup | nothing about skipped apps | skipped 4 volume dumps but `success:true`; torn `.tmp` not promoted; next run honest and complete | 122 s after re-attach | `storage_disconnected` + 4, **all mails suppressed by cooldown** | the second drive loss (R-521); the incomplete run (R-519) | +| F7 | system disk to 95 % | dashboard „90 % · Kritikusan kevés hely"; banner **English** „SSD disk usage high: 90%" | stayed reachable; uploads and a backup still succeeded; warnings cleared 3½ min after cleanup | — | `health_degraded` / true, **mail suppressed by cooldown**; no disk event | operator told nothing (R-521) | diff --git a/documentation/audits/evidence-bignight-2026-09-14/journal.md b/documentation/audits/evidence-bignight-2026-09-14/journal.md index 0b13da2b..74d7ebe8 100644 --- a/documentation/audits/evidence-bignight-2026-09-14/journal.md +++ b/documentation/audits/evidence-bignight-2026-09-14/journal.md @@ -586,3 +586,25 @@ storage page (control „Tárhely" present) — the stale banner cleared within Verdict **PASS for the box** (reachable, nothing refused or broke at 95 %, the backup still completed); **the operator was not told**. + +### F8 — the internet goes away for 20 minutes (`phase5/F8-internet-gone.txt`, `phase5/F8/`) + +**Harness slip, attempt 1 (21:05:19Z):** the bridge-filter chain was named `fwd`, a reserved word in nft; the chain was +never created, so **nothing was cut** (guest → hub still 302). Stopped after 1 minute; evidence kept as +`F8-attempt1-HARNESS-SLIP-rule-not-applied.txt`; the half-made empty table removed. The rule was rewritten as a file, +dry-run checked (`nft -c -f` → OK) and applied for attempt 2. Scope: only `tap333i0` (VM 333); 9201/9202 untouched. + +**Attempt 2 (21:06:25Z):** rule applied, counters 0 → debugged with counting tables: VM 333's frames do pass the bridge +forward hook; the real reason the guest still reached „the internet" is that **`hub.felhom.eu` resolves to +`192.168.0.192` on this LAN** (the hub runs on DooPlex's ingress), so the probe never left the LAN. Stopped; tables removed. + +**Attempt 3 (the measured run), cut at 21:08:36Z** — VM 333 may reach the LAN (DNS 192.168.0.250 / .1, household +devices) but **not** the internet **and not** the hub's LAN ingress 192.168.0.192, which is what a household loses when +its internet goes. The LAN dashboard is probed from demo-hp (a household laptop) instead of DooPlex. Drop counters at +21:09:26Z: 27 packets to the hub, 126 to the internet, 6 IPv6 — the cut is real. Guest probe (fixed script, +`phase5/F8/guest-wan-probe.txt`) at 21:09:48Z: `guest->1.1.1.1=fail guest->hub=fail cloudflared(last120s +registered=1 errors=45)`. (The F8 runner's own guest column prints a shell quoting error — harness; the separate +probe file is the observable.) +Hub at 21:08:42Z (just after the cut): „Last report: 14 min ago · **warn**" — **not a stall**: the controller's +`hub-report` job runs every 15 min (pushed 20:39:46Z and 20:54:46Z, `Hub report pushed successfully`); the „warn" is the +state that last report carried (F7's disk). The next push, due 21:09:46Z, fell inside the cut. diff --git a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8-attempt1-HARNESS-SLIP-rule-not-applied.txt b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8-attempt1-HARNESS-SLIP-rule-not-applied.txt new file mode 100644 index 00000000..6fe82837 --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8-attempt1-HARNESS-SLIP-rule-not-applied.txt @@ -0,0 +1,12 @@ +Error: syntax error, unexpected fwd, expecting string or last +add chain bridge bignight fwd { type filter hook forward priority 0; policy accept; } + ^^^ +Error: syntax error, unexpected policy +add chain bridge bignight fwd { type filter hook forward priority 0; policy accept; } + ^^^^^^ +Error: syntax error, unexpected '}' +add chain bridge bignight fwd { type filter hook forward priority 0; policy accept; } + ^ +F8 internet cut at 21:05:19 +21:05:20 +1s LAN-dashboard=200 guest->internet=302 public-name=502 +21:05:47 +28s LAN-dashboard=200 guest->internet=302 public-name=502 diff --git a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8-internet-gone.txt b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8-internet-gone.txt index 17a7e787..62be4019 100644 --- a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8-internet-gone.txt +++ b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8-internet-gone.txt @@ -1,11 +1,31 @@ -Error: syntax error, unexpected fwd, expecting string or last -add chain bridge bignight fwd { type filter hook forward priority 0; policy accept; } - ^^^ -Error: syntax error, unexpected policy -add chain bridge bignight fwd { type filter hook forward priority 0; policy accept; } - ^^^^^^ -Error: syntax error, unexpected '}' -add chain bridge bignight fwd { type filter hook forward priority 0; policy accept; } - ^ -F8 internet cut at 21:05:19 -21:05:20 +1s LAN-dashboard=200 guest->internet=302 public-name=502 +(attempt 3: rule blocks non-LAN and the hub's LAN ingress; LAN probe from demo-hp) +table bridge bignight { + chain bn_forward { + type filter hook forward priority filter; policy accept; + iifname "tap333i0" ip daddr 192.168.0.192 counter packets 0 bytes 0 drop + iifname "tap333i0" ip daddr 192.168.0.0/24 counter packets 0 bytes 0 accept + iifname "tap333i0" ether type ip counter packets 0 bytes 0 drop + iifname "tap333i0" ether type ip6 counter packets 0 bytes 0 drop + } +} +F8 internet cut at 21:08:36 +21:08:41 +5s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=502 +21:09:07 +31s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:09:33 +57s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:09:59 +83s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:10:25 +109s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:10:51 +135s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:11:18 +162s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:11:44 +188s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:12:10 +214s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:12:36 +240s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 diff --git a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/attempt1-poll-internet-NOT-cut.txt b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/attempt1-poll-internet-NOT-cut.txt new file mode 100644 index 00000000..6470df5e --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/attempt1-poll-internet-NOT-cut.txt @@ -0,0 +1,8 @@ +=== 21:05:51 +hub: Last report: 11 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | warn | Controller | 0.242.0 | Last r +LAN dashboard head: ↗ | + internet -> kumentáció platform | Fut | Megnyitás | Napló | Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Docmost | Moder + Cloudflare -> Egyszerű, könyv-szerű wiki és dokumentáció platform | Fut | Megnyitás | Napló | Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. + hub -> absent + kapcsolat -> ió platform | Fut | Megnyitás | Napló | Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Docmost | Modern wiki é + offline -> absent diff --git a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/guest-wan-probe.txt b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/guest-wan-probe.txt new file mode 100644 index 00000000..43673c44 --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/guest-wan-probe.txt @@ -0,0 +1,2 @@ +21:09:30 +21:11:20 guest->1.1.1.1=fail guest->hub=fail cloudflared(last120s registered=0 errors=15) diff --git a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/hub-and-dashboard-poll.txt b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/hub-and-dashboard-poll.txt new file mode 100644 index 00000000..0ae3d113 --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/hub-and-dashboard-poll.txt @@ -0,0 +1,12 @@ +=== 21:08:42 +hub: Last report: 14 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | warn | Controller | 0.242. +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:10:33 +hub: Last report: 16 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | warn | Controller | 0.242. +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:12:23 +hub: Last report: 18 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | warn | Controller | 0.242. +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc