AI-Based Fraud Detection and Customer Protection in Online Markets: Balancing Transaction Security and Enhancing Digital Payment Adoption

Abstract

The rapid global adoption of digital payments has precipitated an escalating arms race against sophisticated financial fraud, creating a critical tension between implementing robust security measures and maintaining a frictionless user experience that fosters acceptance. Advanced, current, AI-driven fraud detection systems often operate in silos, lack adaptive mechanisms against evolving threats, and fail to explicitly optimise for customer-centric outcomes like trust and satisfaction. To fill these gaps, this study proposes, develops, and validates the Adaptive Security-Trust Equilibrium (ASTE) Framework, a new AI system that aims to find a balance between security and the use of digital payments. The framework synergises a multi-layered transaction risk engine (utilising an extended isolation forest and a stacked ensemble of LightGBM and DNN classifiers), a temporal graph neural network for detecting collusive fraud rings, adaptive behavioural biometrics for continuous authentication, and an Explainable AI (XAI) interface for transparency. Validated on a large-scale amalgamated dataset of over 1.5 million transactions (3.2% fraud rate) synthesised from public and synthetic sources, the ASTE Framework demonstrated superior performance. It achieved a fraud detection F₂ score of 0.891 (recall: 0.924) while reducing the false positive rate to 0.8%, which translated to a 52% reduction in the customer friction index compared to a random forest baseline. The framework extended prior models by formally integrating graph-based relational analysis and a dynamic user trust score into a meta-fusion decision layer. Robustness checks confirmed stable performance under temporal drift and noisy data conditions. Simulations indicated a projected 22.4% increase in median user trust scores and a strong positive correlation (*r* = 0.71) between XAI explanations and user confidence. The findings confirm that a holistic, adaptive, and explainable AI architecture can successfully reconcile the security-adoption paradox, offering a viable path for payment providers to enhance protection, improve customer experience, and support secure financial inclusion. The rapid global adoption of digital payments has precipitated an escalating arms race against sophisticated financial fraud, creating a critical tension between implementing robust security measures and maintaining a frictionless user experience that fosters acceptance. Advanced, current, AI-driven fraud detection systems often operate in silos, lack adaptive mechanisms against evolving threats, and fail to explicitly optimise for customer-centric outcomes like trust and satisfaction. To fill these gaps, this study proposes, develops, and validates the Adaptive Security-Trust Equilibrium (ASTE) Framework, a new AI system that aims to find a balance between security and the use of digital payments. The framework synergises a multi-layered transaction risk engine (utilising an extended isolation forest and a stacked ensemble of LightGBM and DNN classifiers), a temporal graph neural network for detecting collusive fraud rings, adaptive behavioural biometrics for continuous authentication, and an Explainable AI (XAI) interface for transparency. Validated on a large-scale amalgamated dataset of over 1.5 million transactions (3.2% fraud rate) synthesised from public and synthetic sources, the ASTE Framework demonstrated superior performance. It achieved a fraud detection F₂ score of 0.891 (recall: 0.924) while reducing the false positive rate to 0.8%, which translated to a 52% reduction in the customer friction index compared to a random forest baseline. The framework extended prior models by formally integrating graph-based relational analysis and a dynamic user trust score into a meta-fusion decision layer. Robustness checks confirmed stable performance under temporal drift and noisy data conditions. Simulations indicated a projected 22.4% increase in median user trust scores and a strong positive correlation (r = 0.71) between XAI explanations and user confidence. The findings confirm that a holistic, adaptive, and explainable AI architecture can successfully reconcile the security-adoption paradox, offering a viable path for payment providers to enhance protection, improve customer experience, and support secure financial inclusion. The rapid global adoption of digital payments has precipitated an escalating arms race against sophisticated financial fraud, creating a critical tension between implementing robust security measures and maintaining a frictionless user experience that fosters acceptance. Advanced, current, AI-driven fraud detection systems often operate in silos, lack adaptive mechanisms against evolving threats, and fail to explicitly optimise for customer-centric outcomes like trust and satisfaction. To fill these gaps, this study proposes, develops, and validates the Adaptive Security-Trust Equilibrium (ASTE) Framework, a new AI system that aims to find a balance between security and the use of digital payments. The framework synergises a multi-layered transaction risk engine (utilising an extended isolation forest and a stacked ensemble of LightGBM and DNN classifiers), a temporal graph neural network for detecting collusive fraud rings, adaptive behavioural biometrics for continuous authentication, and an Explainable AI (XAI) interface for transparency. Validated on a large-scale amalgamated dataset of over 1.5 million transactions (3.2% fraud rate) synthesised from public and synthetic sources, the ASTE Framework demonstrated superior performance. It achieved a fraud detection F₂ score of 0.891 (recall: 0.924) while reducing the false positive rate to 0.8%, which translated to a 52% reduction in the customer friction index compared to a random forest baseline. The framework extended prior models by formally integrating graph-based relational analysis and a dynamic user trust score into a meta-fusion decision layer. Robustness checks confirmed stable performance under temporal drift and noisy data conditions. Simulations indicated a projected 22.4% increase in median user trust scores and a strong positive correlation (r = 0.71) between XAI explanations and user confidence. The findings confirm that a holistic, adaptive, and explainable AI architecture can successfully reconcile the security-adoption paradox, offering a viable path for payment providers to enhance protection, improve customer experience, and support secure financial inclusion.