37 lines
4.1 KiB
Markdown
37 lines
4.1 KiB
Markdown
# TEST-REPORT — Test campaign #3: NO MERCY (N100 / guest 9201)
|
|
|
|
**Run start:** 2026-06-22 (CC, unattended). Brutal chaos/edge — push to unintended states, verify
|
|
handles-or-fails-safe, then recovers. Every break carries a timed auto-revert; Phase-0 backup is the floor.
|
|
**Legend:** PASS / FAIL / SKIP + raw evidence. Each chaos line: break→detect→recover→verify.
|
|
(Campaigns #1/#2 + diagnoses in git history + `felhom.eu/documentation/tests/`.)
|
|
|
|
## Phase 0 — Baseline + floor — **PASS (gate OPEN)**
|
|
- ctrl **v0.75.0** healthy; agent v0.39.0; **25 containers, 0 unhealthy**; rootfs `/` 4%, `/var/lib/docker` 8%, drives 1%; mem **8.8 Gi available**.
|
|
- Floor: PBS `felhom-pbs:backup/ct/9201/2026-06-22T19:01:56Z` (success, crash-consistent, verified).
|
|
|
|
## Phase 1 — Resource starvation — **PASS (fail-safe held)**
|
|
|
|
| # | Break (timed-revert) | Result | Evidence |
|
|
|---|---|---|---|
|
|
| R1 | python hog ~8 GB (~91% RAM), 75s | **PASS** | mid-stress: 10Gi used / 1.1Gi avail — controller **healthy**, 25/0 unhealthy; post: **dockerd NRestarts=0, no OOM** (alloc fit via cache eviction), recovered to 8.9Gi. Backstop pkill guard armed. |
|
|
| R2 | settings save on full disk | **PASS (code-verified)** | `settings.save()` = atomic write-`.tmp`-then-`os.Rename` (`settings.go:236-261`) → a disk-full fails at WriteFile, original untouched. Live disk-full not reproducible: settings.json is on the 252 GB docker vol. |
|
|
| R3 | fill rootfs (holds `/mnt/sys_drive` backups) to 94% (timed rm @150s) | **PASS** | controller stayed healthy (it's on the separate 252 GB vol); DB-dump + Tier-2 (8 apps, immich 155MB) **succeeded**; **settings.json stayed valid JSON** (no corruption). Reverted → rootfs 4%. (Literal-full refusal not pushed — near-0 rootfs risks wedging the guest OS unattended; the 2 GB margin had room so the gate didn't need to refuse.) |
|
|
| R4 | memory gate hard-block | **PASS (code-verified)** | `deploy.go:186` is a **hard block** (returns an error on `committed+new > usable`), using **committed-memory** accounting (sum of deployed `mem_request`), reserved 384 MB, usable 11904 MB. Live committed ≈ 4.4 GB → ~7.5 GB headroom, so tripping needs mass-deploying ~7.5 GB of requests (impractical/risky unattended); the branch + math are verified. |
|
|
## Phase 2 — State corruption [HIGH SEV] — **mostly PASS; 1 medium finding (S1)**
|
|
|
|
| # | Break (backed up first) | Result | Evidence |
|
|
|---|---|---|---|
|
|
| S1 | truncate settings.json → restart | **⚠ FINDING (medium)** | controller **crash-loops**: `[FATAL] Failed to load settings … unexpected end of JSON input`, RestartCount=7, restarting. **No safe-defaults fallback** — a corrupt settings.json takes the management plane down. *Not silent* (FATAL logged — better than the worst case). Restore → running/healthy, 3 storage_paths back. (Docker restart-manager cycling it re-confirms Finding #1.) |
|
|
| S2 | garbage in uptime-kuma `app.yaml` → rescan | **PASS** | controller stays **healthy** (25/0), logs `[WARN] LoadAppConfig: yaml: … did not find expected key` (not silent, not fatal), other apps unaffected. Restore → `running/deployed`. **Contrast with S1:** per-stack app.yaml corruption is graceful; settings.json is fatal. |
|
|
| S3 | corrupt quiesce marker (bad JSON) → restart | **PASS (minor finding)** | no panic/crash-loop, controller healthy, **stacks not stranded** (restarts=0). But the unparseable marker is **silently ignored** (no log) **and left in place** (not cleared/quarantined) — a real corrupted-mid-quiesce marker would skip recovery with no signal. |
|
|
| S4 | symlink `→ /etc` in userdata/appdata → backup | **PASS (security holds)** 🔒 | Tier-2 rsync **preserved the symlink** (`EVIL_ETC -> /etc`), did **not** follow it; **/etc contents did NOT leak** into the backup (no passwd/shadow). No path-escape/exfil. |
|
|
## Phase 3 — Concurrency storms — _pending_
|
|
## Phase 4 — Time chaos — _pending_
|
|
## Phase 5 — Network partitions — _pending_
|
|
## Phase 6 — Input/security fuzzing [HIGH SEV] — _pending_
|
|
## Phase 7 — Brutal recovery — _pending_
|
|
|
|
---
|
|
## Findings (ranked by severity) — _filled at end_
|
|
## Cleanup confirmation — _filled at end_
|