Search papers, labs, and topics across Lattice.
The paper introduces "Boiling the Frog," a multi-turn benchmark designed to evaluate the susceptibility of tool-using AI agents in corporate/office settings to incremental attacks. The benchmark assesses whether agents, starting from benign states, can be manipulated through a series of interactions to reach unsafe artifact states, using an operational risk taxonomy aligned with the AI Act. Results across nine models show a significant vulnerability, with aggregate attack success rates reaching 44.4% and specific models exceeding 80%, particularly in loss-of-control scenarios.
Even state-of-the-art AI agents are surprisingly vulnerable to incremental attacks that gradually lead them to perform unsafe actions in realistic workplace scenarios.
Background. Traditional safety benchmarks for language models evaluate generated text: whether a model outputs toxic language, reproduces bias, or follows harmful instructions. When models are deployed as agents, the safety-relevant object shifts from what the system says to what it does within an environment, and evaluating model responses under prompting is no longer sufficient to address the safety challenges posed by artificial intelligence. Recent developments have seen the rise of benchmarks that evaluate large language models as agents. We contribute to this strand of research. Approach. We introduce Boiling the Frog, a benchmark that evaluates whether tool-using AI models deployed in corporate and office settings are susceptible to incremental attacks. Each scenario begins with benign workspace edits and later introduces a risk-bearing request. The benchmark focuses on stateful multi-turn evaluation: chains expose a persistent workspace, place the risk-bearing payload at controlled positions in the turn sequence, and score whether the resulting artifact state becomes unsafe. Scenarios are organized through a three-level operational risk taxonomy grounded in the Boiling the Frog risks, the AI Act Annex I and Annex III high-risk contexts, and EU AI Act's Code of Practice on General-Purpose AI (GPAI). Results. Across a nine-model panel, aggregate strict attack success rate (ASR) is 44.4%. Model-level ASR ranges from 20.5% for Claude Haiku 4.5 to 92.9% for Gemini 3.1 Flash Lite, with Seed 2.0 Lite also above 80%. Average chain category-level ASR reaches 93.3% for Code of Practice loss-of-control scenarios.
