Search papers, labs, and topics across Lattice.
This paper introduces AIMS, a novel dataset of 1,724 safety prompts annotated with user intent and harm labels, to evaluate the impact of intent-aware training on safety classifiers. The authors demonstrate that incorporating intent as a signal significantly enhances classifier performance across various training regimes, with intent-conditioned distillation and intent faithfulness rewards yielding the best results. Notably, their approach establishes a Pareto frontier for inference latency and F1 score, underscoring the effectiveness of intent modeling in improving safety classification robustness.
Intent-aware training can dramatically enhance safety classifier performance, revealing that faithful intent modeling is a powerful supervision signal.
We argue that safety classifiers should model user intent as an explicit signal between the prompt and the final label. To study this, we introduce AIMS, a human-annotated dataset of 1,724 difficult safety prompts, each paired with an intent description and harm label. We use AIMS to evaluate intent-aware training across supervised fine-tuning, preference learning, reasoning distillation, and reinforcement learning. Despite its size, AIMS enables competitive safety classifiers across training regimes: DPO from model-generated intent errors improves over SFT, and intent-conditioned distillation outperforms reasoning-only distillation in most teacher-student pairs. Most notably, directly rewarding intent faithfulness with GRPO yields the strongest average performance across five external safety benchmarks, while our intent-aware models form the inference latency-F1 Pareto frontier. These results show that faithful intent modeling is a compact, high-quality supervision signal for more robust safety classifiers.
NVIDIA