From a408256c7eca5f02c634b42df5464aed96ab926c Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 23 Oct 2026 07:39:27 +0200 Subject: [PATCH] REPORT-night-2026-10-23: the night report (five lines, parts 1-5, findings -> verdicts, the test night PASS, teardown in three layers, commits + CI); STATUS: the test night's verdict in the morning list; Nextcloud sha256 + teardown evidence --- REPORT-night-2026-10-23.md | 221 ++++++++++++++++++ STATUS.md | 4 +- .../P4-testnight/18-nextcloud-sha256.txt | 6 + .../P4-testnight/22-teardown.txt | 15 ++ 4 files changed, 244 insertions(+), 2 deletions(-) create mode 100644 REPORT-night-2026-10-23.md create mode 100644 documentation/audits/night-2026-10-23/P4-testnight/18-nextcloud-sha256.txt create mode 100644 documentation/audits/night-2026-10-23/P4-testnight/22-teardown.txt diff --git a/REPORT-night-2026-10-23.md b/REPORT-night-2026-10-23.md new file mode 100644 index 00000000..268efd58 --- /dev/null +++ b/REPORT-night-2026-10-23.md @@ -0,0 +1,221 @@ +# REPORT — night 2026-10-22 → 23: the stray-copy fix, D30/D31 built, a fresh review, one full test night, the DooPlex survey + +Night brief 20:15 → 08:00, unattended. **Nothing was released, deployed, published or vouched.** No change on ep0 or to +DooPlex's configuration. Design read: `07-backup-architecture.md` §6.3, §8 (now §8.2b); `03-host-agent.md` and +`06-offsite-connectivity.md` (the WireGuard peer, the key, the full-list push to ep0); `05-hub-architecture.md` +(re-issue after a host delete, §15.4b); `01-topology-and-trust.md`, `runbooks/total-loss-of-dooplex.md`, +`runbooks/RUNBOOK-hub-db-offsite-backup.md`, `runbooks/gitea-restore.md`, `runbooks/secrets.md`. + +**Safest readings taken (unattended):** (1) D29/D30/D31 are the operator's accepted picks (brief §0); built as picked. +(2) The test box binds to Tester4 with the bind link the hub minted at 2026-10-22 07:37 (unconsumed, valid to 10-29) — +no new link was minted. (3) The test box's root password is read from the hub vault (the break-glass path) because the +day-0 install rotates it. (4) No hub image is built tonight: the morning builds and deploys hub 0.167.0 in one step +(a hub image build in an unattended session is not CC's, `09` §3 decision 162). (5) Review fixes to code that copies or +deletes customer data were made only with a red-proof AND a 9202 proof (rc.8). + +## 0. Five lines (plain words) + +1. **Stray-copy fix proven:** the night copy's container can no longer outlive the controller — removed by name on an + error or timeout, every leftover removed at start, a restore refused while a copy runs; proven twice on the scratch + box (rc.5, and rc.8 after two more review rounds). +2. **D30 and D31 built:** D30 proven live — a box's tunnel that ep0 refused healed itself with a new key in 20 minutes + (and a short clock jump heals by itself, no new key needed); D31 built and red-proved (hub, not deployed). +3. **The review:** four fresh reviews and the commits' own security review — 41 points; every real one is fixed (code, + red-proved; copy code also on the scratch box) or a row; one question for you (D31's reach, morning item 1). +4. **The test night: PASS** — a fresh 8 GB box with the release set, a household and 3.5 h of uploads: at night 0 of 1 833 + health checks failed (worst gap 12 s), every night job ran and passed, nothing uploaded was lost, the 03:30 copy + peaked at 14 % memory stall (39 % on 2026-10-21). One gap: no whole-box copy to ep0 (R-1232, likely an old copy counted). +5. **DooPlex's three biggest gaps:** no printed break-glass sheet (without it no off-site copy opens), the image + registry and agent packages exist only on DooPlex (no update, reinstall or install after a loss), and the website, + installer and hub all hang on the home line. + +## 4. Part 4 — one full test night (Tester4-16fdef, nested VM 9344) + +Installed tonight from ISO 1.30.0; the release set by hand (controller rc.6 → rc.8, agent `0.163.0-night.1` → `0.162.99` += agent main, the D4 drop-in by the bundle's own steps); the rehearsal's 8 apps; household seed 20261023 (3 414 files, +3.5 GB); uploads 20:04–23:38 UTC; the night chain untouched. Evidence: `audits/night-2026-10-23/P4-testnight/` +(`00-provenance.txt` first — it records the extra load below). + +| | Load window 19:56–23:38 UTC | Night 23:38–05:00 UTC | +|---|---|---| +| health probes / failed | 1 175 / 2 (both explained, below) | 1 833 / **0** | +| worst gap between answers | 23 s (22:03, my planned controller swap) | **12 s** (slowest answer 0.27 s) | +| worst memory „full" (10 s, host / guest) | 22.7 % / 24.3 % (1 sample) | 28.2 % / 30.6 % (00:32, the dump leg; 1 sample) | +| samples > 20 % | 2, never two in a row | 4, never two in a row | +| the 03:30 second-drive copy | — | **13.6 % / 17.4 % at most** (2026-10-21: 39.4 % / 45.5 %, ~3 min) | +| OOM kills | 0 | 0 | + +- **Night jobs, all on time, all passed:** dumps 00:30 (3 m 55 s; Immich's model cache „excluded by the template — not + dumped" = D21); second drive 01:30 (8 apps, 4 m 47 s, each copy in its named capped container, **none left after**); + off-site 02:15 (8 apps, 80 snapshots, 5 m 27 s); proof 03:30 PASSED; integrity 04:00 PASSED (100 % re-read). +- **Uploads:** Immich — 90 users, each exactly 303 photos (the set's 306 minus 2 duplicates and 1 empty file refused by + design): **nothing lost** (R-1215 did not recur); Paperless — every upload accepted; Nextcloud — 3 031 of 3 033 files byte-identical to the manifest (sha256; the 2 are the uploader's by-design + skips: the 1.5 GB file over its 1 GB limit and the link to it — the same as 2026-10-21) (`18-nextcloud-sha256.txt`). +- **D4:** root password over SSH refused, root key works, the web page's password login works (console not re-run + tonight; proven 2026-10-22). **D30:** proven on this box before the uploads (Part 2). +- **The two failed probes:** 20:16:34 — the hub judged the box „stuck" while it only waited for the household's + recovery code, re-issued the off-site login, and the box's controller restarted (R-1230, fixed in hub source tonight); + 22:03:40 — my planned switch to rc.8. +- **Extra load, said plainly:** the first (void) load attempt's loop was not stopped (my `pkill -f` killed its own shell + first) and ran beside the planned load 19:56–22:14 — about twice the planned upload work for two hours. The box + carried it without a gap. +- **Not seen tonight:** the whole-box copy to ep0 did not run; most likely the old Tester4 box's ep0 copy (same guest + number) counts as this box's weekly copy — R-1232 (P2, likely, not proven). The local whole-box tier ran at install + (19:20) and is due again after 24 h, correctly. +- Other: one `[ERROR] execCommand failed` line naming nothing (also on 2026-10-21) — R-1233. + + + +## 3. Part 3 — the fresh review: findings → verdicts + +Three read-only reviewers (controller `v0.339.0..main`, agent `v0.162.0..main` + the D4 patches, hub `0.165.0..main` + +the website patch), then a fourth on tonight's own new code, then the commits' automatic security review. Each finding +checked at file:line by the main session. + +| # | Finding | Verdict | +|---|---|---| +| C1 | the capped copy's container outlives the controller | REAL → fixed (Part 1) | +| C2 | two more `.fab` import paths merge old and new data | REAL, pre-existing → R-1187 (recovery code) — not a hold | +| C3 | the import refusal also catches an unhealthy app, wrong advice | REAL → text fixed; the path → R-1187 | +| C4 | `-v` creates a missing path on the daemon side | REAL → `--mount` | +| C5 | the catch-up holds the whole-box backup during its slow checks | REAL → fixed, red-proved | +| C6, C8 | comments / README overstate | REAL → fixed | +| C7 | image tag, no `--pull never` | REAL → fixed; „no memory cgroup": NOT-A-FINDING (every guest is an LXC on a cgroup-v2 Proxmox kernel) | +| A1 | `systemctl reload ssh` fails on a stopped / socket-activated sshd → bundle route frozen | REAL → D4 patch v2 (`try-reload-or-restart`) | +| A2 | nothing checks the drop-in is in force | REAL → v2 reads `sshd -T` back, reports `sshd_dropin_not_in_force` | +| A3 | an undone bundle leaves sshd reloaded with the drop-in | REAL → v2 reloads in undo | +| A4 | downgrade / rollout order | NOT-A-FINDING: bundles go forward by signed jobs only; the morning list ships both halves in one step | +| A5 | the installer swallows a failed reload | REAL → installer patch v2 warns | +| A6, A7 | a wrong citation; „may only ever" wording | REAL → v2 | +| A8 / H1 | after a reboot the future stamp is gone, nothing alarms | REAL → R-1223 | +| H2, H7 | „ok" said to be mailed; WARN for ok | REAL → comment and log level fixed | +| H3 | the operator cooldown is per customer, not per host | REAL → R-1224 | +| H4 | the alarm repeats after the 90-day event prune | NOT-A-FINDING: a reminder every 90 days for a tunnel still dead is wanted; host_swap does the same | +| H5 | a wire-gate entry is false now | REAL → fixed | +| H6 | two reports at once could raise two events | NOT-A-FINDING: the agent reports from one loop, one at a time | +| W1 | the website patch applies, no over-promise | — | +| W2 | 8 GB minimum cannot run Immich | REAL → R-1225 (the operator's wording) | +| W3 | FAQ: data leaves home only encrypted (Cloudflare reads tunnel traffic) | REAL, pre-existing → R-1226 (the operator's wording) | +| N1 | D31's „never made a key" = „held no escrow row" — a deleted install that held a PBS secret is skipped too | REAL as a mechanism → **hold or go** (morning item 1); CC: go | +| N2 | one failed `docker rm` refuses every restore until a restart | REAL → fixed (per-name strays) | +| N3 | the start-up retry removes this process's own copies | REAL → fixed | +| N4 | a second copy of the same destination kills the first | REAL → fixed (one per destination) | +| N5 | `RunTier2` from the „Tier 2 on" save ignores a running restore | REAL, pre-existing → fixed | +| N6 | a power cut during the key rotation can leave no key | REAL → fixed (fsync, link, rename over) | +| N7 | rotation can repeat hourly; the first old key overwritten | REAL → fixed (first key kept, bound saved) | +| N8 | „within about 15 minutes" | REAL → text fixed | +| N9 | the import refusal advises removing an unhealthy app | REAL → text fixed | +| S1 | (security review) the start-up removal could hit a copy starting between list and remove | REAL → fixed (the destination's lock) | +| S2 | (security review) a restore beginning between the copy's check and its record | REAL → fixed (one step under the lock; every leg asks again) | +| S3 | (security review) a wait stamp dated after now blocks the re-key | REAL → fixed | + +## 1. Part 1 — the stray-copy fix + +The capped second-drive copy (R-1214) ran `rsync -a --delete` in its own container; killing the `docker` CLI (the +60-minute limit, a restart, a self-update) left the container deleting outside every guard. Controller `66a69d4`, +`912e592`, then after the reviews `943c517` (C5), `9d9e404` (N2–N5, N9), `1babfe9` (two races): a fixed name + label; +removed by name on any error; every leftover removed at start before the scheduler and the web server; a restore +refused while a copy runs or may run; one copy per destination at a time; `RunTier2` refuses during a restore for every +caller, in one step with its record; `--mount`, the image ID, `--pull never`, `--no-healthcheck`. +**Proof on 9202:** rc.5 and again rc.8 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB +copy → the old container removed at start, by name, logged; a restore probe during the copy refused; the redone copy +equal to the source; a 15-second test build → the timed-out container removed, the partial copy stopped growing; two +runs into the same destination → the second waited, no kill, no failure. 9202 back on 0.339.0. + +## 2. Part 2 — D30, D31, D29 + +- **D30** (agent `ae69ad4`, hardened `39fb9cb`, `b9b831f`; hub `84751cb0`, `c6649dd4`): live on the test box + (`P2-D30/`). Run 1: a 2-hour clock jump healed by itself in 3 minutes — ep0 started a handshake (it had traffic for the + box) — and the agent correctly did not re-key. Run 2 (ep0's handshake starts dropped on the box, the rehearsal's + condition): dead tunnel → 15 min → **RE-KEYED** → the hub took the new key in place (`existed=false generation=3 + sync=ok`, same 10.77.0.5) → the box switched at its next desired-state fetch → a handshake with the new key, tunnel + ping ok. The live hub (0.165.0) stored the alarm's input (`handshake_future_s=6851`); the alarm and its clearing are + pinned by hub tests (0.167.0, not deployed). **Nothing new on ep0:** the hub's existing full-list push. +- **D31** (hub `84751cb0`): the F-14 re-issue reads the newest deletion of a host that held a key; red-proved. The + review's N1 is the morning's hold-or-go question. +- **D29**: recorded (CONTEXT, R-925). + +## 5. Part 5 — the DooPlex survey (read only) + +`documentation/audits/dooplex-survival-2026-10-22/PLAN.md`: every thing on DooPlex with where it lives, what copies it, +where the copy is, who can restore it, how long — measured from unit states, journals, sizes, pods, DNS. Both nightly +off-site jobs and both weekly restore tests are green. `total-loss-of-dooplex.md` marked step by step (works today / +works once / never tried; one stale line fixed: the mail hold shipped in hub 0.145.0). Ten gaps, each with two options and +CC's pick; four go to the operator as D32–D35. **The three biggest:** the paper sheet (without S1 no off-site copy +opens), the registry and agent packages (only on DooPlex — no update, reinstall or install after a loss), and the home +line (website, installer, hub — new installs stop). + +## 4. Records + +**Commits.** +- felhom-controller: `66a69d4`, `912e592`, `943c517`, `9d9e404`, `1babfe9`, `23f77f9` (docs). +- felhom-agent: `ae69ad4`, `39fb9cb`, `b9b831f`, `b45c47b` (REPORT). +- felhom.eu: `84751cb0` (hub D30/D31 + Part 1 evidence), `75854432` (D30 proof, D4 v2, survey, 03/05/07, CONTEXT), + `c6649dd4` (hub text), `70a600d0` (register), `1805f8bb` (STATUS), + the report commit. +- **CI by commit (Gitea jobs, all success, attempt 1):** controller `66a69d4` 2227, `912e592` 2228, `943c517` 2231, + `9d9e404` 2233, `1babfe9` 2234, `23f77f9` 2240; agent `ae69ad4` 2229, `39fb9cb` 2235, `b9b831f` 2237, `b45c47b` 2241; + felhom.eu `84751cb0` 2230, `75854432` 2232, `c6649dd4` 2236, `1805f8bb` 2238 (with `70a600d0`), `e94d0256` 2239 (with + `14388e21`), `5dc17cab` 2242 (with `f56f3328`), `7802da72` 2243 (with `1cb3e571`) — a commit pushed with a later one + has no job of its own; the last report commit's job is checked after this file. + +**Images.** Controller `0.340.0-rc.5` (= `66a69d4`), `rc.6` (= `943c517`), `rc.7` (= `9d9e404`, built, not used: the +security review came first), `rc.8` (= `1babfe9`). No hub image (the morning builds 0.167.0 with its deploy). A +test-only controller image (rc.5 + a 15 s limit) lived only on DooPlex and 9202 and was removed from both. + +**Dated checks due today, done:** R-32 (Tester4's listing 72 → 80, none vanished), R-1173 (the other outcome). + +**Rows:** before 180, after 191. Opened 11: R-1223 (the tunnel reboot gap), R-1224 (the per-customer cooldown), R-1225 and +R-1226 (two website sentences, D36), R-1227 (the registry, D33), R-1228 (the home line, D35), R-1229 (ArgoCD settings), +R-1230 (a held box reported stuck — fixed in hub source), R-1231 (the recovery page says „older"), R-1232 (the first ep0 +copy of a reinstalled box), R-1233 (an error line naming nothing). Closed 0 — the night's fixes close on release +(R-1214, R-1204, R-1205, R-1230 are READY). Updated: R-1214, R-1204, R-1205, R-925, R-1187, R-982, R-232, R-923, R-924. + +**Teardown, three layers.** +- **Machine:** 9202 back on 0.339.0, healthy, test files and the test image removed, no copy container (rc.5 / rc.8 stay + in its own image store, as earlier rc images). The test box: VM 9344 destroyed with its disks, after every log was + copied off (`P4-testnight/13`–`21`). +- **Host (demo-hp):** VM 9344 and loader CT 9345 destroyed (own IDs; disks were on nvme-scratch, never the thin pool); + `/tmp` clean. Standing items untouched: VM 341, CT 9401, the ISOs in `local` (see R-548). +- **Hub:** host **Tester4-16fdef** stays registered (it will read „down") — **the operator deletes it** on the Hosts page, + with appliance 49 (morning item 6). Tester4's off-site store holds tonight's 8 new snapshots (80 in all; left, as on + 2026-10-21); its PBS token was re-issued automatically at the install (F-14, product behaviour); its WireGuard peer + holds the D30 key until the host is deleted. +- **DooPlex:** nothing changed — the survey read only; builds and the scratchpad only; the local test image removed. + +**System poster:** no listed fact changed (no machine, time, tier or custody moved; the WireGuard key rotation is not a +poster fact) — none of the three actions needed. **Instruction files:** none edited. + +**Observations.** +- demo-hp (a standing box): its local whole-box tier was refused 4 times tonight by the agent's space check (8.5 GB free, + 9.2 GB needed), and the controller stopped 11 apps for about a minute before each try — FILED: R-548 (updated; the + superseded installer ISOs on `local` are the easy half). +- The capped copy inherited the controller's health check („starting") — NOT-A-FINDING: small, fixed in the session + (`--no-healthcheck`, `912e592`). +- One outside health check answered 502 at 20:16:34Z: the hub's false „stuck" re-issue restarted the controller — + FILED: R-1230 (fixed in hub source tonight). +- The Nextcloud tree upload takes ~3 h under the load (same as 2026-10-21: 18:53 → 21:56). — NOT-A-FINDING: the same + load gives the same time; not a regression. +- The soak tools assumed the setup gate takes the dashboard cookie; the gate now hands out its own cookie + (`felhom_gate`) through `/__gate/start`. — NOT-A-FINDING: the product works as designed; the tool was updated. +- The first load attempt (19:55Z) got 404/401 everywhere and was stopped after 2 minutes, logs kept as `void-1955/` + in the loader. — NOT-A-FINDING: the tool's gate cookie (above); nothing on the box changed. +- The VOID 9202 run's restore probe started a restore of a nonexistent app, failed at once. — NOT-A-FINDING: no data; + explained in `P1-straycopy/README.md`. +- A test request's error printed one character of a test password into the session (a Python traceback). — + NOT-A-FINDING for the product; the probe now encodes the password and prints nothing of it. +- The off-site unlock was refused once: the agent fixture's pre-release version string („0.163.0-night.1") is not a + version the controller can compare, so its capability read „unknown" — NOT-A-FINDING for the product (no pre-release + agent is ever released; the fixture was rebuilt as 0.162.99); the page's wording for „unknown" — FILED: R-1231. +- The household's off-site snapshots carry the rehearsal's future date (2026-10-28) — NOT-A-FINDING: known, R-1206; + retention was not due tonight, so its case did not come up. +- The night copy sat at the guest's memory limit while it ran (77 845 limit events, 0 OOM) with almost no stall — NOT-A- + FINDING: the capped copy's cache is reclaimed inside the guest by design; the consequence (pressure, health) is the + measure, and it improved. +- **CC's own mistakes:** (1) a commit (`70a600d0`) was made while its fast gate failed — the command chained the commit + after printing the gate's code; the push hook refused it, STATUS was fixed, and both commits went up together with + green gates. (2) Two gate runs at once (mine and the push hook's) made the decoy test fail; it passes alone. (3) The + void load loop ran on for 2 h 18 min because my `pkill -f load2.py` matched its own shell (a known trap); found at + 22:14, stopped by pid, recorded in the provenance; the same trap hit once more at the teardown (a `pkill -f` killed my own + command; the teardown was rerun). (4) The first agent fixture used a version string the controller + cannot compare, which cost one refused unlock. + diff --git a/STATUS.md b/STATUS.md index 34ebcc5c..e7df5bc1 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night is running and its verdict lands here at 07:00. Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35). +**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night PASSED (no failed check at night, every night job passed, nothing lost). Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35). ## What works - A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4). @@ -49,7 +49,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12. **Morning, 2026-10-23 — in this order.** -1. Hold or go: The test night is running; its verdict lands here at 07:00. One question from the review: D31 as built also covers an install that got the whole-box password but no recovery package — its password is rotated without your press (R-1205). CC: go. If not: revert one hub commit before item 4. +1. Hold or go: CC says go — the test night passed on a fresh 8 GB box (no failed check at night, every night job passed, nothing lost; one gap: no whole-box copy to ep0, R-1232). One question from the review: D31 as built also covers an install that got the whole-box password but no recovery package — its password is rotated without your press (R-1205). CC: go. If not: revert one hub commit before item 4. 2. Release controller 0.340.0 (main after tonight; tested as rc.8 on the scratch box, rc.6 on the test box): the safe night copy, the catch-up fix, R-49, R-1187, R-1189, R-1214. If not: these wait. 3. Release the agent (the tunnel repair, its tunnel field, the root-SSH bundle) and the installer tag with the root-SSH removal (R-1204, R-982); then vouch. If not: a clock-ahead box stays cut off; root keeps its password on the LAN. 4. Build and deploy hub 0.167.0 (D31, the tunnel alarm and its clearing). If not: a dead tunnel stays silent. diff --git a/documentation/audits/night-2026-10-23/P4-testnight/18-nextcloud-sha256.txt b/documentation/audits/night-2026-10-23/P4-testnight/18-nextcloud-sha256.txt new file mode 100644 index 00000000..08bcd51f --- /dev/null +++ b/documentation/audits/night-2026-10-23/P4-testnight/18-nextcloud-sha256.txt @@ -0,0 +1,6 @@ +== Csalad_feltoltes +nextcloud Csalad_feltoltes: manifest 3033 files; on the box 3031; byte-identical 3031; missing/different 2; extra 0 + MISSING /Csalad_feltoltes/nagy_fajl_1_5GB.bin + MISSING /Csalad_feltoltes/Kulonleges_nevek/link_a_nagy_fajlra +== Csalad_proof1 +(Csalad_proof1: not finished — stopped at 05:38 UTC for the teardown; the verdict rests on Csalad_feltoltes, the full tree) diff --git a/documentation/audits/night-2026-10-23/P4-testnight/22-teardown.txt b/documentation/audits/night-2026-10-23/P4-testnight/22-teardown.txt new file mode 100644 index 00000000..c5486fd5 --- /dev/null +++ b/documentation/audits/night-2026-10-23/P4-testnight/22-teardown.txt @@ -0,0 +1,15 @@ +== teardown 2026-10-23T05:37:39Z — demo-hp, own IDs only (VM 9344, CT 9345) +name: night1023-soak +hostname: night1023-loader +purging VM 9344 from related configurations.. +purging CT 9345 from related configurations.. +after: + VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID + 341 night1004-tester1 running 8192 200.00 1860 +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness +no 9344/9345 images left +vmshot-9344.ppm +/tmp/vmshot-9344.ppm removed