Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of Large Language Models (LLMs) in detecting online scams without the need for task-specific fine-tuning, utilizing a newly curated benchmark dataset of real-world scams. The research reveals that while larger LLMs typically outperform smaller models, effective prompting can significantly enhance the performance of smaller models, allowing them to generalize better to unseen scams compared to fine-tuned classifiers. The findings underscore the importance of pre-trained knowledge in scam detection, providing valuable insights for the integration of LLMs into security products.
LLMs can outperform fine-tuned classifiers in scam detection, particularly when smaller models are effectively prompted, revealing a surprising potential for broader applications in security.
Online scams continue to cause substantial financial and personal harm. As a result, detection systems based on Large Language Models (LLMs) have been integrated into security products ranging from email gateways and browser extensions to fraud-monitoring dashboards. As this adoption accelerates, a common belief has taken hold: that these models are broadly suitable for scam detection. In this work, we investigate whether LLMs, with their strong capabilities in understanding intent, context, and reasoning, can effectively detect scams across diverse scenarios without task-specific fine-tuning. We curate and release a unique benchmark dataset of real-world scams spanning multiple formats and topics. We evaluate nine LLMs of varying sizes and architectures, examining their performance under different prompting strategies and comparing them to a fine-tuned BERT-based classifier. Our results show that while larger LLMs generally outperform smaller ones, effective prompting substantially boosts the performance of smaller models. Moreover, LLMs are better at generalizing to unseen scams compared to fine-tuned models, suggesting that pre-trained knowledge contributes meaningfully to scam detection. We release our dataset and evaluation framework to facilitate future research in robust scam detection using language models.