Enhancing Financial Security through the Integration of Machine Learning Models for Effective Fraud Detection in Transaction Systems

Fraud detection (FD) in financial transactions involves identifying and preventing unauthorized or suspicious activities to safeguard financial systems and customer assets. However, traditional fraud detection approaches in financial transactions often fall short due to their reliance on predefined rules and static thresholds, which limits their capability to adapt to evolving fraud patterns and detect sophisticated or emerging threats effectively. To address the challenges in traditional FD methods, this paper proposes an advanced machine learning-based model, Enhancing Financial Security through the Integration of Machine Learning methods for Effective Fraud Detection in Transaction Systems (EFS-IML-EFD-TS). Initially, input data is gathered from the Financial Fraud Detection Dataset. The collected data is first pre-processed utilizing the Confidence Partitioning Sampling Filtering (CPSF) technique to handle missing values, remove duplicate records, and standardize feature scaling. The pre-processed data is further processed using the Exponential Distance Transform (EDT), which extracts discriminative features like transaction amount, time of day, and location.Then, the imbalanced data is balanced using Adaptive Support Vector-Borderline SMOTE (ASV-SMOTE), which generates high-quality synthetic samples near decision boundaries, reducing noise and improving minority class prediction. Then Interpretable Generalized Additive Neural Network (IGANN) is used to detect fraud and classify financial transactions as either genuine or fraudulent. The proposed EFS-IML-EFD-TS method achieves 98.5% precision, 98% accuracy, 97% recall, 97.5% F1-score, 0.91 MCC, a high AUC of 0.9636, low loss of 0.05, and the shortest computational time of 1.125 seconds, compared with existing methods such as Online payment fraud detection model utilizing machine learning techniques (OPFT-MLT-ANN),Financial Fraud Detection utilizing Value-at-Risk with Machine Learning in Skewed Data (FFD-MLSD-DNN), and Transparency and privacy: the role of explainable AI and federated learning in financial fraud detection (TP-AI-FFD-DNN).