Search papers, labs, and topics across Lattice.
This study investigates a layered fraud detection pipeline using the PaySim dataset, integrating a gradient-boosted classifier, graph features, an autoencoder for anomaly detection, and an investigation agent for uncertain cases. Despite initial expectations, the graph features and anomaly signals did not enhance overall performance but showed improved ranking for fraud cases within a specific score range. The investigation agent, while providing coherent rationales, ultimately underperformed compared to direct classifier thresholding, highlighting the nuanced contributions of each system component and the potential pitfalls of relying on model explanations alone.
An investigation agent can misinterpret classifier outputs, leading to more errors despite providing seemingly rational explanations.
Fraud detection systems must scale with rising transaction volume while remaining explainable and reviewable. We study a layered pipeline on the PaySim dataset that combines a gradient-boosted classifier, graph-derived structural features, an autoencoder-based anomaly signal, TreeSHAP explanations, and a bounded LLM investigation agent applied to cases the classifier scores uncertainly. Before any model comparison, we identify and remove a simulator-specific balance shortcut that would otherwise inflate baseline performance. After this correction, neither the graph features nor the anomaly signal improves Average Precision on the full test set. Both, however, rank fraud better within the subset of cases receiving intermediate baseline scores. In a controlled experiment with injected multi-account fraud rings, engineered structural features recover all injected test transactions, while the tabular baseline misses roughly a quarter of them. The investigation agent underperforms direct thresholding of the classifier it relies on, reaching 65.0% accuracy against 71.7% on a balanced 60-case sample, despite having access to model explanations, graph context, and retrieved reference cases. Of the eight decisions the agent changed, six replaced correct classifier outputs with errors, and it produced a coherent written rationale in each case. An exploratory disagreement-based escalation rule flagged two of these agent errors for human review without flagging any correct decision. We conclude that each component of a layered fraud system contributes only under specific conditions, and that a plausible rationale from an investigation agent is not evidence of a better decision.
