Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces TF-RefusalBench, a multilingual benchmark designed to assess over-alignment in LLMs specifically for criminal law applications, derived from Swiss Federal Supreme Court rulings. The research reveals that over-alignment is a complex issue influenced by both the model and the languages involved, highlighting that traditional metrics like over-refusal do not adequately capture the impact on task fidelity. By evaluating various mitigation strategies, the authors find that abliteration effectively reduces refusals without significantly compromising task performance, offering a practical solution for deploying LLMs in sensitive legal contexts.
Over-alignment in LLMs can severely hinder their utility in criminal law, but a novel approach shows how to reduce refusals while maintaining performance.
While the wider applicability of LLMs in the legal field is currently debated due to their reliability and the gravity of any errors, narrow uses with well-understood and mitigated risks have emerged. Notably the Swiss Federal Supreme Court uses small on-premises models for tentative translations and short-passage summarization across the four official languages. However, such usage is challenging in the context of Criminal Law. Since rulings and cases employees work on routinely can contain detailed descriptions of violent and sexual offenses, their legitimate work is compromised by refusals and disclaimers due to the activation of model guardrails (over-alignment). To measure this phenomenon, we introduce TF-RefusalBench, a multilingual benchmark for criminal-law translation and summarization derived from public Swiss Supreme Court rulings. TF-RefusalBench contains 5,200 total prompts across French, German, Italian, and English, corresponding to common task prompts and passages likely to trigger refusal. We then use TF-RefusalBench to show that over-alignment is a multifaceted phenomenon, influenced by the model and the prompt and text languages being processed, and that its impact cannot be evaluated solely from an over-refusal perspective, given the disclaimer's impact on task faithfulness. Finally, we evaluate approaches to enable on-premises LLMs for Criminal Law Tasks, demonstrating that while prompting can be effective, abliteration (refusal directions ablation) eliminates refusal with minimal impact on task performance.