Search papers, labs, and topics across Lattice.
This study evaluates the reasoning capabilities of OpenAI's GPT 5.4 in the context of legal case forecasting, specifically using cases from the European Court of Human Rights. The researchers explored various prompting strategies to assess how well the model performs in generating legally meaningful reasoning, finding that while the model produces structurally complete analyses, these are often substantively shallow and misaligned with human evaluators. The results highlight the limitations of relying on LLMs for legal reasoning tasks, emphasizing the need for human oversight in evaluation processes.
LLMs can generate structurally sound legal analyses but often fall short in substantive reasoning, raising questions about their reliability in legal contexts.
Reasoning has become a standard technique and feature for contemporary LLMs; however, its application and quality in the context of demanding legal-oriented tasks, such as legal case forecasting, remain under explored. We investigate how LLMs reason in the context of legal case forecasting, using legal cases from the European Court of Human Rights (ECtHR) as a testbed. We evaluate OpenAI GPT 5.4, a recent top-tier LLM, by exploring alternative prompting strategies that are more or less suggestive of what counts as legally meaningful reasoning in the context of ECtHR jurisprudence. We present our findings derived from assessing the model's responses with both human and LLM evaluation. We find that the examined model scores far from ideal in legal reasoning, the model produces structurally complete but substantively shallow analyses, and that LLM-as-a-Judge evaluators are internally consistent yet align only weakly with our trained annotators, i.e., reliable but not a valid substitute for human evaluation. Overall, the expert-curated prompt leads to more comprehensive reasoning, which does not result in more accurate predictions compared to the other examined settings. Based on our findings, we urge the community not to rely solely on automated LLM-based evaluation and to avoid using task accuracy as an appropriate proxy for reasoning quality.
