Search papers, labs, and topics across Lattice.
The paper introduces TraceSafe-Bench, a benchmark to evaluate the safety of LLMs during multi-step tool-use trajectories, encompassing 12 risk categories. Experiments on 13 LLM-as-a-guard models and 7 specialized guardrails revealed that guardrail efficacy is more dependent on structural data competence (e.g., JSON parsing) than semantic safety alignment, and that model architecture is more influential than model size. Surprisingly, risk detection accuracy improves across longer trajectories as models adapt to dynamic execution behaviors.
LLM safety guardrails are surprisingly more about parsing JSON than preventing jailbreaks, and general-purpose LLMs often outperform specialized safety models in multi-step tool-use scenarios.
As large language models (LLMs) evolve from static chatbots into autonomous agents, the primary vulnerability surface shifts from final outputs to intermediate execution traces. While safety guardrails are well-benchmarked for natural language responses, their efficacy remains largely unexplored within multi-step tool-use trajectories. To address this gap, we introduce TraceSafe-Bench, the first comprehensive benchmark specifically designed to assess mid-trajectory safety. It encompasses 12 risk categories, ranging from security threats (e.g., prompt injection, privacy leaks) to operational failures (e.g., hallucinations, interface inconsistencies), featuring over 1,000 unique execution instances. Our evaluation of 13 LLM-as-a-guard models and 7 specialized guardrails yields three critical findings: 1) Structural Bottleneck: Guardrail efficacy is driven more by structural data competence (e.g., JSON parsing) than semantic safety alignment. Performance correlates strongly with structured-to-text benchmarks ($蟻=0.79$) but shows near-zero correlation with standard jailbreak robustness. 2) Architecture over Scale: Model architecture influences risk detection performance more significantly than model size, with general-purpose LLMs consistently outperforming specialized safety guardrails in trajectory analysis. 3) Temporal Stability: Accuracy remains resilient across extended trajectories. Increased execution steps allow models to pivot from static tool definitions to dynamic execution behaviors, actually improving risk detection performance in later stages. Our findings suggest that securing agentic workflows requires jointly optimizing for structural reasoning and safety alignment to effectively mitigate mid-trajectory risks.
