Search papers, labs, and topics across Lattice.
This paper introduces Safety Sentry, a context-aware guard model that enhances decision-making for LLM agents by framing the safety assessment as a three-way routing problem鈥擡XECUTE, ASK, or REFUSE鈥攔ather than a binary safe/unsafe classification. By allowing for context-specific evaluations, Safety Sentry reduces unnecessary interruptions and improves user autonomy while maintaining high accuracy and safety-related recall. The model achieves superior performance compared to existing baselines, demonstrating its effectiveness in managing risk in real-world applications of LLMs.
Safety Sentry redefines safety in LLM interactions by enabling context-sensitive decision-making, significantly reducing user interruptions while enhancing safety outcomes.
LLM agents act on real-world environments through tool calls, and a single misjudged action can cause irreversible harm. The standard safeguard is a guard model that labels each proposed action as safe or unsafe, but this binary view conflates two distinct decisions: whether the action is harmful in itself, and whether it is appropriate given the user's context. It also operates at the granularity of action categories rather than individual instances, producing routine interruptions that erode autonomy and train users to wave through the most consequential alerts. We reframe the problem as a per-instance three-way routing decision over {EXECUTE, ASK, REFUSE} and instantiate it with Safety Sentry, a lightweight guard model whose inference reduces to a single decoding call. A single decoding-time threshold lets one fixed checkpoint be re-positioned across deployments of differing risk tolerance without retraining. Safety Sentry outperforms a broad set of open-weight and frontier closed-source baselines on overall accuracy and safety-related recall, while controlling both directional error rates simultaneously.
