Search papers, labs, and topics across Lattice.
This paper identifies a critical vulnerability in stateful, function-calling environments of large language models (LLMs) that traditional prompt-centric defenses overlook. By employing a black-box attack framework called Simulated Moderation Traces (SMT), the authors demonstrate how adversarial inputs can exploit the blurred boundaries between trusted control logic and untrusted data, leading to harmful outputs. Empirical evaluations reveal that SMT significantly outperforms existing attack methods, achieving the highest success rates while requiring minimal queries, underscoring the inadequacy of prompt-level defenses alone.
Adversaries can exploit structural vulnerabilities in function-calling LLMs to bypass safety measures, achieving high success rates with minimal effort.
Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs). While prior work has primarily studied attacks and defenses at the prompt level, we show that this prompt-centric paradigm overlooks a structural vulnerability in stateful, function-calling environments. In such applications, developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context. This architecture expands the attack surface by blurring the boundary between trusted control logic and untrusted data, allowing adversarial intent to be distributed across a multi-turn execution path. We exploit this architectural flaw through SMT, a black-box attack framework based on Simulated Moderation Traces. Departing from purely prompt-based interactions, SMT constructs a multi-turn trajectory that simulates a legitimate moderation-auditing workflow. Within this trajectory, a fabricated moderation frame leverages red-team testing as a pretext to elicit harmful generations. The subsequent validation feedback treats safety refusals as execution failures, prompting refinements that gradually weaken the model's safety constraints and ultimately trigger harmful outputs. Extensive empirical evaluations on prominent commercial LLMs from five different providers across two standardized safety benchmarks show that SMT consistently achieves the highest average attack success rate and HarmScore while requiring a near-minimal number of queries, substantially outperforming existing baselines. These findings demonstrate that prompt-level sanitization alone is fundamentally insufficient for defending tool-enabled LLM systems and highlight the urgent need for context-aware validation across schemas, arguments, tool outputs, and accumulated conversation state. The code is available at https://github.com/liujlong27/SMT.
