Data mining and Machine learning are emerging technologies rapidly spreading in every field of life due to their beneficial aspects. The financial sector also makes use of these technologies. Many studies on banking data analysis have been performed using machine learning techniques. These research studies also have many Problems as the main focus of these studies was to achieve high accuracy, and some only perform a comparative analysis of different classifiers' performance. Another major drawback of these studies was that they did not identify optimal parameters and their impact. In this research, we have identified optimal parameters. These parameters are valuable for credit scoring and might also be used to predict credit card defaulters. We also find their impact on the results. We have used feature selection and classification techniques to identify optimal parameters and their effect on credit card defaulter identification. We have introduced three classifiers, Kstar, SMO, and Multilayer perceptron, and repeated the classification and feature selection process for every classifier. First, we apply feature selection techniques to our dataset with each classifier to find possible optimal parameters. In the next phase, we use classification to see the impact of possible optimal parameters and prove our findings. In each classification round, we used different parameters available in the dataset (included and excluded) and noted each classification's results. This way, we identify the optimal parameters and their impact on the results. Whereas we also analyze the performance of classifiers. To perform this research study, we use the "credit card defaults" dataset, which we obtained from UCI Machine learning online repository. We use two feature selection techniques: the ranker approach and the evolutionary search method, and after that, we also apply classification techniques to the dataset. This research can help to reduce the complexities of the credit scoring process. Through this study, we identify up to six optimal parameters and find their impact on the performance of classifiers. Further, We also identified that multilayer perceptron was the best-performing classifier out of the three. This research work can also be extended to other fields in the future where we use this mechanism to find optimal parameters, and their impact can help us predict the results.