Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Guardian-as-an-Advisor (GaaA), a soft-gating safety mechanism where a guardian model predicts risk and provides explanations, which are then prepended to the original query for re-inference by the base model. To train and evaluate this approach, the authors construct GuardSet, a 208k+ dataset designed to unify harmful and harmless cases while focusing on robustness and honesty. Experiments show that GuardAdvisor achieves competitive detection accuracy, improves response quality when used to augment inputs, and introduces minimal latency overhead.
Soft-gating with an "advisor" model can steer LLMs to be safer and more useful, reducing over-refusal without sacrificing detection accuracy.
Hard-gated safety checkers often over-refuse and misalign with a vendor's model spec; prevailing taxonomies also neglect robustness and honesty, yielding safer-on-paper yet less useful systems. This work introduces Guardian-as-an-Advisor (GaaA), a soft-gating pipeline where a guardian predicts a binary risk label plus a concise explanation and prepends this advice to the original query for re-inference, keeping the base model operating under its original spec. To support training and evaluation, GuardSet is constructed, a 208k+ multi-domain dataset unifying harmful and harmless cases with targeted robustness and honesty slices. GuardAdvisor is trained via SFT followed by RL to enforce label-explanation consistency. GuardAdvisor attains competitive detection accuracy while enabling the advisory workflow; when used to augment inputs, responses improve over unaugmented prompts. A latency study shows advisor inference uses below 5% of base-model compute and adds only 2-10% end-to-end overhead under realistic harmful-input rates. Overall, GaaA steers models to comply with the model spec, maintaining safety while reducing over-refusal.
MIT CSAIL