REPORT-night-2026-10-23: the night report (five lines, parts 1-5, findings -> verdicts, the test night PASS, teardown in three layers, commits + CI); STATUS: the test night's verdict in the morning list; Nextcloud sha256 + teardown evidence

REPORT-night-2026-10-23.md

@@ -0,0 +1,221 @@
+# REPORT — night 2026-10-22 → 23: the stray-copy fix, D30/D31 built, a fresh review, one full test night, the DooPlex survey
+
+Night brief 20:15 → 08:00, unattended. **Nothing was released, deployed, published or vouched.** No change on ep0 or to
+DooPlex's configuration. Design read: `07-backup-architecture.md` §6.3, §8 (now §8.2b); `03-host-agent.md` and
+`06-offsite-connectivity.md` (the WireGuard peer, the key, the full-list push to ep0); `05-hub-architecture.md`
+(re-issue after a host delete, §15.4b); `01-topology-and-trust.md`, `runbooks/total-loss-of-dooplex.md`,
+`runbooks/RUNBOOK-hub-db-offsite-backup.md`, `runbooks/gitea-restore.md`, `runbooks/secrets.md`.
+
+**Safest readings taken (unattended):** (1) D29/D30/D31 are the operator's accepted picks (brief §0); built as picked.
+(2) The test box binds to Tester4 with the bind link the hub minted at 2026-10-22 07:37 (unconsumed, valid to 10-29) —
+no new link was minted. (3) The test box's root password is read from the hub vault (the break-glass path) because the
+day-0 install rotates it. (4) No hub image is built tonight: the morning builds and deploys hub 0.167.0 in one step
+(a hub image build in an unattended session is not CC's, `09` §3 decision 162). (5) Review fixes to code that copies or
+deletes customer data were made only with a red-proof AND a 9202 proof (rc.8).
+
+## 0. Five lines (plain words)
+
+1. **Stray-copy fix proven:** the night copy's container can no longer outlive the controller — removed by name on an
+   error or timeout, every leftover removed at start, a restore refused while a copy runs; proven twice on the scratch
+   box (rc.5, and rc.8 after two more review rounds).
+2. **D30 and D31 built:** D30 proven live — a box's tunnel that ep0 refused healed itself with a new key in 20 minutes
+   (and a short clock jump heals by itself, no new key needed); D31 built and red-proved (hub, not deployed).
+3. **The review:** four fresh reviews and the commits' own security review — 41 points; every real one is fixed (code,
+   red-proved; copy code also on the scratch box) or a row; one question for you (D31's reach, morning item 1).
+4. **The test night: PASS** — a fresh 8 GB box with the release set, a household and 3.5 h of uploads: at night 0 of 1 833
+   health checks failed (worst gap 12 s), every night job ran and passed, nothing uploaded was lost, the 03:30 copy
+   peaked at 14 % memory stall (39 % on 2026-10-21). One gap: no whole-box copy to ep0 (R-1232, likely an old copy counted).
+5. **DooPlex's three biggest gaps:** no printed break-glass sheet (without it no off-site copy opens), the image
+   registry and agent packages exist only on DooPlex (no update, reinstall or install after a loss), and the website,
+   installer and hub all hang on the home line.
+
+## 4. Part 4 — one full test night (Tester4-16fdef, nested VM 9344)
+
+Installed tonight from ISO 1.30.0; the release set by hand (controller rc.6 → rc.8, agent `0.163.0-night.1` → `0.162.99`
+= agent main, the D4 drop-in by the bundle's own steps); the rehearsal's 8 apps; household seed 20261023 (3 414 files,
+3.5 GB); uploads 20:04–23:38 UTC; the night chain untouched. Evidence: `audits/night-2026-10-23/P4-testnight/`
+(`00-provenance.txt` first — it records the extra load below).
+
+| | Load window 19:56–23:38 UTC | Night 23:38–05:00 UTC |
+|---|---|---|
+| health probes / failed | 1 175 / 2 (both explained, below) | 1 833 / **0** |
+| worst gap between answers | 23 s (22:03, my planned controller swap) | **12 s** (slowest answer 0.27 s) |
+| worst memory „full" (10 s, host / guest) | 22.7 % / 24.3 % (1 sample) | 28.2 % / 30.6 % (00:32, the dump leg; 1 sample) |
+| samples > 20 % | 2, never two in a row | 4, never two in a row |
+| the 03:30 second-drive copy | — | **13.6 % / 17.4 % at most** (2026-10-21: 39.4 % / 45.5 %, ~3 min) |
+| OOM kills | 0 | 0 |
+
+- **Night jobs, all on time, all passed:** dumps 00:30 (3 m 55 s; Immich's model cache „excluded by the template — not
+  dumped" = D21); second drive 01:30 (8 apps, 4 m 47 s, each copy in its named capped container, **none left after**);
+  off-site 02:15 (8 apps, 80 snapshots, 5 m 27 s); proof 03:30 PASSED; integrity 04:00 PASSED (100 % re-read).
+- **Uploads:** Immich — 90 users, each exactly 303 photos (the set's 306 minus 2 duplicates and 1 empty file refused by
+  design): **nothing lost** (R-1215 did not recur); Paperless — every upload accepted; Nextcloud — 3 031 of 3 033 files byte-identical to the manifest (sha256; the 2 are the uploader's by-design
+  skips: the 1.5 GB file over its 1 GB limit and the link to it — the same as 2026-10-21) (`18-nextcloud-sha256.txt`).
+- **D4:** root password over SSH refused, root key works, the web page's password login works (console not re-run
+  tonight; proven 2026-10-22). **D30:** proven on this box before the uploads (Part 2).
+- **The two failed probes:** 20:16:34 — the hub judged the box „stuck" while it only waited for the household's
+  recovery code, re-issued the off-site login, and the box's controller restarted (R-1230, fixed in hub source tonight);
+  22:03:40 — my planned switch to rc.8.
+- **Extra load, said plainly:** the first (void) load attempt's loop was not stopped (my `pkill -f` killed its own shell
+  first) and ran beside the planned load 19:56–22:14 — about twice the planned upload work for two hours. The box
+  carried it without a gap.
+- **Not seen tonight:** the whole-box copy to ep0 did not run; most likely the old Tester4 box's ep0 copy (same guest
+  number) counts as this box's weekly copy — R-1232 (P2, likely, not proven). The local whole-box tier ran at install
+  (19:20) and is due again after 24 h, correctly.
+- Other: one `[ERROR] execCommand failed` line naming nothing (also on 2026-10-21) — R-1233.
+
+
+
+## 3. Part 3 — the fresh review: findings → verdicts
+
+Three read-only reviewers (controller `v0.339.0..main`, agent `v0.162.0..main` + the D4 patches, hub `0.165.0..main` +
+the website patch), then a fourth on tonight's own new code, then the commits' automatic security review. Each finding
+checked at file:line by the main session.
+
+| # | Finding | Verdict |
+|---|---|---|
+| C1 | the capped copy's container outlives the controller | REAL → fixed (Part 1) |
+| C2 | two more `.fab` import paths merge old and new data | REAL, pre-existing → R-1187 (recovery code) — not a hold |
+| C3 | the import refusal also catches an unhealthy app, wrong advice | REAL → text fixed; the path → R-1187 |
+| C4 | `-v` creates a missing path on the daemon side | REAL → `--mount` |
+| C5 | the catch-up holds the whole-box backup during its slow checks | REAL → fixed, red-proved |
+| C6, C8 | comments / README overstate | REAL → fixed |
+| C7 | image tag, no `--pull never` | REAL → fixed; „no memory cgroup": NOT-A-FINDING (every guest is an LXC on a cgroup-v2 Proxmox kernel) |
+| A1 | `systemctl reload ssh` fails on a stopped / socket-activated sshd → bundle route frozen | REAL → D4 patch v2 (`try-reload-or-restart`) |
+| A2 | nothing checks the drop-in is in force | REAL → v2 reads `sshd -T` back, reports `sshd_dropin_not_in_force` |
+| A3 | an undone bundle leaves sshd reloaded with the drop-in | REAL → v2 reloads in undo |
+| A4 | downgrade / rollout order | NOT-A-FINDING: bundles go forward by signed jobs only; the morning list ships both halves in one step |
+| A5 | the installer swallows a failed reload | REAL → installer patch v2 warns |
+| A6, A7 | a wrong citation; „may only ever" wording | REAL → v2 |
+| A8 / H1 | after a reboot the future stamp is gone, nothing alarms | REAL → R-1223 |
+| H2, H7 | „ok" said to be mailed; WARN for ok | REAL → comment and log level fixed |
+| H3 | the operator cooldown is per customer, not per host | REAL → R-1224 |
+| H4 | the alarm repeats after the 90-day event prune | NOT-A-FINDING: a reminder every 90 days for a tunnel still dead is wanted; host_swap does the same |
+| H5 | a wire-gate entry is false now | REAL → fixed |
+| H6 | two reports at once could raise two events | NOT-A-FINDING: the agent reports from one loop, one at a time |
+| W1 | the website patch applies, no over-promise | — |
+| W2 | 8 GB minimum cannot run Immich | REAL → R-1225 (the operator's wording) |
+| W3 | FAQ: data leaves home only encrypted (Cloudflare reads tunnel traffic) | REAL, pre-existing → R-1226 (the operator's wording) |
+| N1 | D31's „never made a key" = „held no escrow row" — a deleted install that held a PBS secret is skipped too | REAL as a mechanism → **hold or go** (morning item 1); CC: go |
+| N2 | one failed `docker rm` refuses every restore until a restart | REAL → fixed (per-name strays) |
+| N3 | the start-up retry removes this process's own copies | REAL → fixed |
+| N4 | a second copy of the same destination kills the first | REAL → fixed (one per destination) |
+| N5 | `RunTier2` from the „Tier 2 on" save ignores a running restore | REAL, pre-existing → fixed |
+| N6 | a power cut during the key rotation can leave no key | REAL → fixed (fsync, link, rename over) |
+| N7 | rotation can repeat hourly; the first old key overwritten | REAL → fixed (first key kept, bound saved) |
+| N8 | „within about 15 minutes" | REAL → text fixed |
+| N9 | the import refusal advises removing an unhealthy app | REAL → text fixed |
+| S1 | (security review) the start-up removal could hit a copy starting between list and remove | REAL → fixed (the destination's lock) |
+| S2 | (security review) a restore beginning between the copy's check and its record | REAL → fixed (one step under the lock; every leg asks again) |
+| S3 | (security review) a wait stamp dated after now blocks the re-key | REAL → fixed |
+
+## 1. Part 1 — the stray-copy fix
+
+The capped second-drive copy (R-1214) ran `rsync -a --delete` in its own container; killing the `docker` CLI (the
+60-minute limit, a restart, a self-update) left the container deleting outside every guard. Controller `66a69d4`,
+`912e592`, then after the reviews `943c517` (C5), `9d9e404` (N2–N5, N9), `1babfe9` (two races): a fixed name + label;
+removed by name on any error; every leftover removed at start before the scheduler and the web server; a restore
+refused while a copy runs or may run; one copy per destination at a time; `RunTier2` refuses during a restore for every
+caller, in one step with its record; `--mount`, the image ID, `--pull never`, `--no-healthcheck`.
+**Proof on 9202:** rc.5 and again rc.8 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB
+copy → the old container removed at start, by name, logged; a restore probe during the copy refused; the redone copy
+equal to the source; a 15-second test build → the timed-out container removed, the partial copy stopped growing; two
+runs into the same destination → the second waited, no kill, no failure. 9202 back on 0.339.0.
+
+## 2. Part 2 — D30, D31, D29
+
+- **D30** (agent `ae69ad4`, hardened `39fb9cb`, `b9b831f`; hub `84751cb0`, `c6649dd4`): live on the test box
+  (`P2-D30/`). Run 1: a 2-hour clock jump healed by itself in 3 minutes — ep0 started a handshake (it had traffic for the
+  box) — and the agent correctly did not re-key. Run 2 (ep0's handshake starts dropped on the box, the rehearsal's
+  condition): dead tunnel → 15 min → **RE-KEYED** → the hub took the new key in place (`existed=false generation=3
+  sync=ok`, same 10.77.0.5) → the box switched at its next desired-state fetch → a handshake with the new key, tunnel
+  ping ok. The live hub (0.165.0) stored the alarm's input (`handshake_future_s=6851`); the alarm and its clearing are
+  pinned by hub tests (0.167.0, not deployed). **Nothing new on ep0:** the hub's existing full-list push.
+- **D31** (hub `84751cb0`): the F-14 re-issue reads the newest deletion of a host that held a key; red-proved. The
+  review's N1 is the morning's hold-or-go question.
+- **D29**: recorded (CONTEXT, R-925).
+
+## 5. Part 5 — the DooPlex survey (read only)
+
+`documentation/audits/dooplex-survival-2026-10-22/PLAN.md`: every thing on DooPlex with where it lives, what copies it,
+where the copy is, who can restore it, how long — measured from unit states, journals, sizes, pods, DNS. Both nightly
+off-site jobs and both weekly restore tests are green. `total-loss-of-dooplex.md` marked step by step (works today /
+works once / never tried; one stale line fixed: the mail hold shipped in hub 0.145.0). Ten gaps, each with two options and
+CC's pick; four go to the operator as D32–D35. **The three biggest:** the paper sheet (without S1 no off-site copy
+opens), the registry and agent packages (only on DooPlex — no update, reinstall or install after a loss), and the home
+line (website, installer, hub — new installs stop).
+
+## 4. Records
+
+**Commits.**
+- felhom-controller: `66a69d4`, `912e592`, `943c517`, `9d9e404`, `1babfe9`, `23f77f9` (docs).
+- felhom-agent: `ae69ad4`, `39fb9cb`, `b9b831f`, `b45c47b` (REPORT).
+- felhom.eu: `84751cb0` (hub D30/D31 + Part 1 evidence), `75854432` (D30 proof, D4 v2, survey, 03/05/07, CONTEXT),
+  `c6649dd4` (hub text), `70a600d0` (register), `1805f8bb` (STATUS), + the report commit.
+- **CI by commit (Gitea jobs, all success, attempt 1):** controller `66a69d4` 2227, `912e592` 2228, `943c517` 2231,
+  `9d9e404` 2233, `1babfe9` 2234, `23f77f9` 2240; agent `ae69ad4` 2229, `39fb9cb` 2235, `b9b831f` 2237, `b45c47b` 2241;
+  felhom.eu `84751cb0` 2230, `75854432` 2232, `c6649dd4` 2236, `1805f8bb` 2238 (with `70a600d0`), `e94d0256` 2239 (with
+  `14388e21`), `5dc17cab` 2242 (with `f56f3328`), `7802da72` 2243 (with `1cb3e571`) — a commit pushed with a later one
+  has no job of its own; the last report commit's job is checked after this file.
+
+**Images.** Controller `0.340.0-rc.5` (= `66a69d4`), `rc.6` (= `943c517`), `rc.7` (= `9d9e404`, built, not used: the
+security review came first), `rc.8` (= `1babfe9`). No hub image (the morning builds 0.167.0 with its deploy). A
+test-only controller image (rc.5 + a 15 s limit) lived only on DooPlex and 9202 and was removed from both.
+
+**Dated checks due today, done:** R-32 (Tester4's listing 72 → 80, none vanished), R-1173 (the other outcome).
+
+**Rows:** before 180, after 191. Opened 11: R-1223 (the tunnel reboot gap), R-1224 (the per-customer cooldown), R-1225 and
+R-1226 (two website sentences, D36), R-1227 (the registry, D33), R-1228 (the home line, D35), R-1229 (ArgoCD settings),
+R-1230 (a held box reported stuck — fixed in hub source), R-1231 (the recovery page says „older"), R-1232 (the first ep0
+copy of a reinstalled box), R-1233 (an error line naming nothing). Closed 0 — the night's fixes close on release
+(R-1214, R-1204, R-1205, R-1230 are READY). Updated: R-1214, R-1204, R-1205, R-925, R-1187, R-982, R-232, R-923, R-924.
+
+**Teardown, three layers.**
+- **Machine:** 9202 back on 0.339.0, healthy, test files and the test image removed, no copy container (rc.5 / rc.8 stay
+  in its own image store, as earlier rc images). The test box: VM 9344 destroyed with its disks, after every log was
+  copied off (`P4-testnight/13`–`21`).
+- **Host (demo-hp):** VM 9344 and loader CT 9345 destroyed (own IDs; disks were on nvme-scratch, never the thin pool);
+  `/tmp` clean. Standing items untouched: VM 341, CT 9401, the ISOs in `local` (see R-548).
+- **Hub:** host **Tester4-16fdef** stays registered (it will read „down") — **the operator deletes it** on the Hosts page,
+  with appliance 49 (morning item 6). Tester4's off-site store holds tonight's 8 new snapshots (80 in all; left, as on
+  2026-10-21); its PBS token was re-issued automatically at the install (F-14, product behaviour); its WireGuard peer
+  holds the D30 key until the host is deleted.
+- **DooPlex:** nothing changed — the survey read only; builds and the scratchpad only; the local test image removed.
+
+**System poster:** no listed fact changed (no machine, time, tier or custody moved; the WireGuard key rotation is not a
+poster fact) — none of the three actions needed. **Instruction files:** none edited.
+
+**Observations.**
+- demo-hp (a standing box): its local whole-box tier was refused 4 times tonight by the agent's space check (8.5 GB free,
+  9.2 GB needed), and the controller stopped 11 apps for about a minute before each try — FILED: R-548 (updated; the
+  superseded installer ISOs on `local` are the easy half).
+- The capped copy inherited the controller's health check („starting") — NOT-A-FINDING: small, fixed in the session
+  (`--no-healthcheck`, `912e592`).
+- One outside health check answered 502 at 20:16:34Z: the hub's false „stuck" re-issue restarted the controller —
+  FILED: R-1230 (fixed in hub source tonight).
+- The Nextcloud tree upload takes ~3 h under the load (same as 2026-10-21: 18:53 → 21:56). — NOT-A-FINDING: the same
+  load gives the same time; not a regression.
+- The soak tools assumed the setup gate takes the dashboard cookie; the gate now hands out its own cookie
+  (`felhom_gate`) through `/__gate/start`. — NOT-A-FINDING: the product works as designed; the tool was updated.
+- The first load attempt (19:55Z) got 404/401 everywhere and was stopped after 2 minutes, logs kept as `void-1955/`
+  in the loader. — NOT-A-FINDING: the tool's gate cookie (above); nothing on the box changed.
+- The VOID 9202 run's restore probe started a restore of a nonexistent app, failed at once. — NOT-A-FINDING: no data;
+  explained in `P1-straycopy/README.md`.
+- A test request's error printed one character of a test password into the session (a Python traceback). —
+  NOT-A-FINDING for the product; the probe now encodes the password and prints nothing of it.
+- The off-site unlock was refused once: the agent fixture's pre-release version string („0.163.0-night.1") is not a
+  version the controller can compare, so its capability read „unknown" — NOT-A-FINDING for the product (no pre-release
+  agent is ever released; the fixture was rebuilt as 0.162.99); the page's wording for „unknown" — FILED: R-1231.
+- The household's off-site snapshots carry the rehearsal's future date (2026-10-28) — NOT-A-FINDING: known, R-1206;
+  retention was not due tonight, so its case did not come up.
+- The night copy sat at the guest's memory limit while it ran (77 845 limit events, 0 OOM) with almost no stall — NOT-A-
+  FINDING: the capped copy's cache is reclaimed inside the guest by design; the consequence (pressure, health) is the
+  measure, and it improved.
+- **CC's own mistakes:** (1) a commit (`70a600d0`) was made while its fast gate failed — the command chained the commit
+  after printing the gate's code; the push hook refused it, STATUS was fixed, and both commits went up together with
+  green gates. (2) Two gate runs at once (mine and the push hook's) made the decoy test fail; it passes alone. (3) The
+  void load loop ran on for 2 h 18 min because my `pkill -f load2.py` matched its own shell (a known trap); found at
+  22:14, stopped by pid, recorded in the provenance; the same trap hit once more at the teardown (a `pkill -f` killed my own
+  command; the teardown was rerun). (4) The first agent fixture used a version string the controller
+  cannot compare, which cost one refused unlock.
+

STATUS.md

@@ -2,7 +2,7 @@
 
 **Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
 
-**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night is running and its verdict lands here at 07:00. Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35).
+**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night PASSED (no failed check at night, every night job passed, nothing lost). Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35).
 
 ## What works
 - A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4).
@@ -49,7 +49,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
 
 **Morning, 2026-10-23 — in this order.**
-1. Hold or go: The test night is running; its verdict lands here at 07:00. One question from the review: D31 as built also covers an install that got the whole-box password but no recovery package — its password is rotated without your press (R-1205). CC: go. If not: revert one hub commit before item 4.
+1. Hold or go: CC says go — the test night passed on a fresh 8 GB box (no failed check at night, every night job passed, nothing lost; one gap: no whole-box copy to ep0, R-1232). One question from the review: D31 as built also covers an install that got the whole-box password but no recovery package — its password is rotated without your press (R-1205). CC: go. If not: revert one hub commit before item 4.
 2. Release controller 0.340.0 (main after tonight; tested as rc.8 on the scratch box, rc.6 on the test box): the safe night copy, the catch-up fix, R-49, R-1187, R-1189, R-1214. If not: these wait.
 3. Release the agent (the tunnel repair, its tunnel field, the root-SSH bundle) and the installer tag with the root-SSH removal (R-1204, R-982); then vouch. If not: a clock-ahead box stays cut off; root keeps its password on the LAN.
 4. Build and deploy hub 0.167.0 (D31, the tunnel alarm and its clearing). If not: a dead tunnel stays silent.

documentation/audits/night-2026-10-23/P4-testnight/18-nextcloud-sha256.txt

@@ -0,0 +1,6 @@
+== Csalad_feltoltes
+nextcloud Csalad_feltoltes: manifest 3033 files; on the box 3031; byte-identical 3031; missing/different 2; extra 0
+    MISSING /Csalad_feltoltes/nagy_fajl_1_5GB.bin
+    MISSING /Csalad_feltoltes/Kulonleges_nevek/link_a_nagy_fajlra
+== Csalad_proof1
+(Csalad_proof1: not finished — stopped at 05:38 UTC for the teardown; the verdict rests on Csalad_feltoltes, the full tree)

documentation/audits/night-2026-10-23/P4-testnight/22-teardown.txt

@@ -0,0 +1,15 @@
+== teardown 2026-10-23T05:37:39Z — demo-hp, own IDs only (VM 9344, CT 9345)
+name: night1023-soak
+hostname: night1023-loader
+purging VM 9344 from related configurations..
+purging CT 9345 from related configurations..
+after:
+      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID       
+       341 night1004-tester1    running    8192             200.00 1860      
+VMID       Status     Lock         Name                
+9201       running                 demo-hp             
+9202       running                 demo-hp-scratch     
+9401       stopped                 upgrade-harness     
+no 9344/9345 images left
+vmshot-9344.ppm
+/tmp/vmshot-9344.ppm removed