Search papers, labs, and topics across Lattice.
The authors developed a multi-turn evaluation framework, grounded in collaboration with law enforcement and policy experts, to assess the potential of large language models (LLMs) in assisting fraud and cybercrime across three scenarios: romance scams, CEO impersonation, and identity theft. They measured the models' ability to provide actionable information by decomposing complex requests into sequences of benign queries, comparing the results to a standard web search baseline. The study found that current LLMs offer limited practical assistance, but open-weight models without safety guardrails and benign query decomposition significantly increased the models' helpfulness in these scenarios.
LLMs are surprisingly bad at helping with fraud and cybercrime, but cleverly disguised prompts and the removal of safety guardrails can significantly boost their criminal utility.
AI is increasingly being used to assist fraud and cybercrime. However, it is unclear whether current large language models can assist complex criminal activity. Working with law enforcement and policy experts, we developed multi-turn evaluations for three fraud and cybercrime scenarios (romance scams, CEO impersonation, and identity theft). Our evaluations focused on text-to-text model capabilities. In each scenario, we measured model capabilities in ways designed to resemble real-world misuse, such as breaking down requests for fraud into a sequence of seemingly benign queries, and measuring whether models provide actionable information, relative to a standard web search baseline. We found that (1) current large language models provide minimal practical assistance with complex criminal activity, (2) open-weight large language models fine-tuned to remove safety guardrails provided substantially more help, and (3) decomposing requests into benign-seeming queries elicited more assistance than explicitly malicious framing or system-level jailbreaks. Overall, the results suggest that current risks from text-generation models are relatively minimal. However, this work contributes a reproducible, expert-grounded framework for tracking how these risks may evolve with time as models grow more capable and adversaries adapt.
