Search papers, labs, and topics across Lattice.
This paper investigates the potential of AI models to simulate justice-specific questioning in moot court exercises using a dataset of U.S. Supreme Court oral argument transcripts. They introduce a two-layer evaluation framework that assesses the realism and pedagogical usefulness of simulated questions, using proxy metrics to capture desirable qualities like anticipating legal issues and detecting weaknesses. Experiments with prompt-based and agentic simulators reveal that while AI-generated questions can be realistic and recall substantive legal issues, they suffer from low diversity and sycophancy, highlighting the need for nuanced evaluation.
AI can generate realistic legal questions, but current models still struggle with diversity and a tendency to agree too much, revealing critical gaps in their ability to simulate adversarial legal reasoning.
In oral arguments, judges probe attorneys with questions about the factual record, legal claims, and the strength of their arguments. To prepare for this questioning, both law schools and practicing attorneys rely on moot courts: practice simulations of appellate hearings. Leveraging a dataset of U.S. Supreme Court oral argument transcripts, we examine whether AI models can effectively simulate justice-specific questioning for moot court-style training. Evaluating oral argument simulation is challenging because there is no single correct question for any given turn. Instead, effective questioning should reflect a combination of desirable qualities, such as anticipating substantive legal issues, detecting logical weaknesses, and maintaining an appropriately adversarial tone. We introduce a two-layer evaluation framework that assesses both the realism and pedagogical usefulness of simulated questions using complementary proxy metrics. We construct and evaluate both prompt-based and agentic oral argument simulators. We find that simulated questions are often perceived as realistic by human annotators and achieve high recall of ground truth substantive legal issues. However, models still face substantial shortcomings, including low diversity in question types and sycophancy. Importantly, these shortcomings would remain undetected under naive evaluation approaches.
Stanford HAI