30 lines
2.6 KiB
Markdown
30 lines
2.6 KiB
Markdown
# TEST-REPORT — Test campaign #3: NO MERCY (N100 / guest 9201)
|
|
|
|
**Run start:** 2026-06-22 (CC, unattended). Brutal chaos/edge — push to unintended states, verify
|
|
handles-or-fails-safe, then recovers. Every break carries a timed auto-revert; Phase-0 backup is the floor.
|
|
**Legend:** PASS / FAIL / SKIP + raw evidence. Each chaos line: break→detect→recover→verify.
|
|
(Campaigns #1/#2 + diagnoses in git history + `felhom.eu/documentation/tests/`.)
|
|
|
|
## Phase 0 — Baseline + floor — **PASS (gate OPEN)**
|
|
- ctrl **v0.75.0** healthy; agent v0.39.0; **25 containers, 0 unhealthy**; rootfs `/` 4%, `/var/lib/docker` 8%, drives 1%; mem **8.8 Gi available**.
|
|
- Floor: PBS `felhom-pbs:backup/ct/9201/2026-06-22T19:01:56Z` (success, crash-consistent, verified).
|
|
|
|
## Phase 1 — Resource starvation — **PASS (fail-safe held)**
|
|
|
|
| # | Break (timed-revert) | Result | Evidence |
|
|
|---|---|---|---|
|
|
| R1 | python hog ~8 GB (~91% RAM), 75s | **PASS** | mid-stress: 10Gi used / 1.1Gi avail — controller **healthy**, 25/0 unhealthy; post: **dockerd NRestarts=0, no OOM** (alloc fit via cache eviction), recovered to 8.9Gi. Backstop pkill guard armed. |
|
|
| R2 | settings save on full disk | **PASS (code-verified)** | `settings.save()` = atomic write-`.tmp`-then-`os.Rename` (`settings.go:236-261`) → a disk-full fails at WriteFile, original untouched. Live disk-full not reproducible: settings.json is on the 252 GB docker vol. |
|
|
| R3 | fill rootfs (holds `/mnt/sys_drive` backups) to 94% (timed rm @150s) | **PASS** | controller stayed healthy (it's on the separate 252 GB vol); DB-dump + Tier-2 (8 apps, immich 155MB) **succeeded**; **settings.json stayed valid JSON** (no corruption). Reverted → rootfs 4%. (Literal-full refusal not pushed — near-0 rootfs risks wedging the guest OS unattended; the 2 GB margin had room so the gate didn't need to refuse.) |
|
|
| R4 | memory gate hard-block | **PASS (code-verified)** | `deploy.go:186` is a **hard block** (returns an error on `committed+new > usable`), using **committed-memory** accounting (sum of deployed `mem_request`), reserved 384 MB, usable 11904 MB. Live committed ≈ 4.4 GB → ~7.5 GB headroom, so tripping needs mass-deploying ~7.5 GB of requests (impractical/risky unattended); the branch + math are verified. |
|
|
## Phase 2 — State corruption [HIGH SEV] — _pending_
|
|
## Phase 3 — Concurrency storms — _pending_
|
|
## Phase 4 — Time chaos — _pending_
|
|
## Phase 5 — Network partitions — _pending_
|
|
## Phase 6 — Input/security fuzzing [HIGH SEV] — _pending_
|
|
## Phase 7 — Brutal recovery — _pending_
|
|
|
|
---
|
|
## Findings (ranked by severity) — _filled at end_
|
|
## Cleanup confirmation — _filled at end_
|