Files
felhom-controller/TEST-REPORT.md
T
2026-06-22 21:16:45 +02:00

2.6 KiB

TEST-REPORT — Test campaign #3: NO MERCY (N100 / guest 9201)

Run start: 2026-06-22 (CC, unattended). Brutal chaos/edge — push to unintended states, verify handles-or-fails-safe, then recovers. Every break carries a timed auto-revert; Phase-0 backup is the floor. Legend: PASS / FAIL / SKIP + raw evidence. Each chaos line: break→detect→recover→verify. (Campaigns #1/#2 + diagnoses in git history + felhom.eu/documentation/tests/.)

Phase 0 — Baseline + floor — PASS (gate OPEN)

  • ctrl v0.75.0 healthy; agent v0.39.0; 25 containers, 0 unhealthy; rootfs / 4%, /var/lib/docker 8%, drives 1%; mem 8.8 Gi available.
  • Floor: PBS felhom-pbs:backup/ct/9201/2026-06-22T19:01:56Z (success, crash-consistent, verified).

Phase 1 — Resource starvation — PASS (fail-safe held)

# Break (timed-revert) Result Evidence
R1 python hog ~8 GB (~91% RAM), 75s PASS mid-stress: 10Gi used / 1.1Gi avail — controller healthy, 25/0 unhealthy; post: dockerd NRestarts=0, no OOM (alloc fit via cache eviction), recovered to 8.9Gi. Backstop pkill guard armed.
R2 settings save on full disk PASS (code-verified) settings.save() = atomic write-.tmp-then-os.Rename (settings.go:236-261) → a disk-full fails at WriteFile, original untouched. Live disk-full not reproducible: settings.json is on the 252 GB docker vol.
R3 fill rootfs (holds /mnt/sys_drive backups) to 94% (timed rm @150s) PASS controller stayed healthy (it's on the separate 252 GB vol); DB-dump + Tier-2 (8 apps, immich 155MB) succeeded; settings.json stayed valid JSON (no corruption). Reverted → rootfs 4%. (Literal-full refusal not pushed — near-0 rootfs risks wedging the guest OS unattended; the 2 GB margin had room so the gate didn't need to refuse.)
R4 memory gate hard-block PASS (code-verified) deploy.go:186 is a hard block (returns an error on committed+new > usable), using committed-memory accounting (sum of deployed mem_request), reserved 384 MB, usable 11904 MB. Live committed ≈ 4.4 GB → ~7.5 GB headroom, so tripping needs mass-deploying ~7.5 GB of requests (impractical/risky unattended); the branch + math are verified.

Phase 2 — State corruption [HIGH SEV] — pending

Phase 3 — Concurrency storms — pending

Phase 4 — Time chaos — pending

Phase 5 — Network partitions — pending

Phase 6 — Input/security fuzzing [HIGH SEV] — pending

Phase 7 — Brutal recovery — pending


Findings (ranked by severity) — filled at end

Cleanup confirmation — filled at end