Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces FraudSMSWalker, a benchmark designed to evaluate the ability of agentic large language models to detect SMS-to-webpage fraud without relying on URL and domain cues. By providing a controlled dataset of 699 bilingual SMS chains, including both fraudulent and benign cases, the study reveals that while models can identify suspicious signals, they often fail to maintain high recall for benign cases, leading to weakly supported positive predictions. The findings highlight the limitations of current models in making accurate and evidence-based fraud judgments when traditional reputation shortcuts are removed.
Current models can spot fraud signals but struggle to distinguish benign cases, risking over-prediction in high-stakes scenarios.
SMS fraud is increasingly cross-channel: a message directs the user to a webpage, and the final risk depends on how the SMS claim aligns with the page content and requested user action. However, existing evaluations either focus on message-only smishing classification or expose URL and domain cues that allow models to rely on reputation shortcuts. To address this gap, we introduce \textbf{FraudSMSWalker}, a controlled benchmark for URL-masked SMS-to-webpage fraud judgment. FraudSMSWalker contains 699 bilingual chains, including 332 fraudulent and 367 benign cases, across ten service scenarios. The model-visible input consists of the SMS context and sanitized webpage evidence, while raw URLs, hosts, domains, IPs, redirects, and reputation metadata are withheld. The benchmark further includes hard benign cases whose pages contain login, payment, verification, or account-management elements that are plausible under the service context but also appear in scam flows. We evaluate nine web agents under masked browser-agent protocols and conduct URL-visibility ablations. The results show that current agents can detect suspicious cues, but struggle to preserve benign recall and often produce positive predictions that are weakly supported by the observed evidence. These findings position FraudSMSWalker as a benchmark for measuring whether web agents can make fraud judgments that remain both accurate and evidence-grounded when direct reputation shortcuts are suppressed. The associated code and dataset are accessible at the \href{https://anonymous.4open.science/w/FraudMessageWalker-Bench}{anonymous link}.