Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of zero-shot prompting of decoder-only models compared to supervised fine-tuning of encoder-only models for ranking case-law sentences relevant to statutory terms. By leveraging a dataset of 26,959 sentences categorized by explanatory value, the authors demonstrate that prompting significantly outperforms fine-tuning approaches across all evaluated concepts and NDCG cutoffs. The findings reveal that the best-performing prompting method surpasses all previously reported state-of-the-art results, highlighting its potential for improving legal information retrieval.
Prompting decoder-only models outperforms fine-tuning methods, achieving unprecedented effectiveness in ranking case-law sentences for statutory term retrieval.
Legal concepts in statutes are often expressed using vague terms, and practitioners frequently turn to case law to interpret them. We study the task of ranking case-law sentences by their usefulness for explaining a concept or target statutory term, using an established dataset of 26,959 sentences covering 42 U.S. Code concepts labeled into four explanatory-value categories. We compare two families of methods: (i) supervised fine-tuning of encoder-only models (ModernBERT) and (ii) zero-shot prompting of decoder-only models. We show that across all concepts and standard NDCG cutoffs, ModernBERT largely matches earlier BERT-family baselines. In contrast, prompting decoder-only models achieves the strongest overall effectiveness, with our best system surpassing all previously reported state-of-the-art results on this task.
CMU ML