Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
LegalBench-BR, a new benchmark for Brazilian legal text classification, was created using LLM-assisted labeling and heuristic validation on 3,105 appellate proceedings across five legal areas. Experiments reveal that general-purpose LLMs like GPT-4o mini and Claude 3.5 Haiku perform poorly, exhibiting systematic biases towards civil law. However, fine-tuning BERTimbau-LoRA on this dataset achieves significantly higher accuracy (87.6%) and macro-F1 score (0.87), demonstrating the necessity of domain-adapted models for Brazilian legal NLP.
GPT-4o mini gets an F1 score of 0.0 on administrative law in Brazil, but a small, fine-tuned BERT model nails it, revealing a critical blind spot for general-purpose LLMs in legal domains.
We introduce LegalBench-BR, the first public benchmark for evaluating language models on Brazilian legal text classification. The dataset comprises 3,105 appellate proceedings from the Santa Catarina State Court (TJSC), collected via the DataJud API (CNJ) and annotated across five legal areas through LLM-assisted labeling with heuristic validation. On a class-balanced test set, BERTimbau-LoRA, updating only 0.3% of model parameters, achieves 87.6% accuracy and 0.87 macro-F1 (+22pp over Claude 3.5 Haiku, +28pp over GPT-4o mini). The gap is most striking on administrativo (administrative law): GPT-4o mini scores F1 = 0.00 and Claude 3.5 Haiku scores F1 = 0.08 on this class, while the fine-tuned model reaches F1 = 0.91. Both commercial LLMs exhibit a systematic bias toward civel (civil law), absorbing ambiguous classes rather than discriminating them, a failure mode that domain-adapted fine-tuning eliminates. These results demonstrate that general-purpose LLMs cannot substitute for domain-adapted models in Brazilian legal classification, even when the task is a simple 5-class problem, and that LoRA fine-tuning on a consumer GPU closes the gap at zero marginal inference cost. We release the full dataset, model, and pipeline to enable reproducible research in Portuguese legal NLP.