Search papers, labs, and topics across Lattice.
This paper introduces a "Complexity Score" algorithm to predict when detailed prompts improve circumstance extraction from death investigation narratives, focusing on inferentially complex cases. They compared LLMs (GPT-5.2, Gemini 2.5 Pro, Llama-3 70B) against fine-tuned RoBERTa on 25 such circumstances from the NVDRS. Results show LLMs significantly outperform fine-tuned models on low-prevalence circumstances, supporting a hybrid architecture.
LLMs crush fine-tuned models on rare, inferentially complex circumstance extraction from death investigation narratives, suggesting a new hybrid approach.
Suicide is a leading cause of death in the United States, and understanding the circumstances that precede it requires extracting structured information from death investigation narratives. Many of these circumstances require semantic inference beyond simple keyword matching. We develop a ``Complexity Score'' algorithm that analyzes coding manual structure to predict when detailed prompts with full coding guidelines improve over name-only prompts. We then construct a hybrid approach that selects prompt strategy per circumstance. We evaluate large language models (LLMs) against fine-tuned RoBERTa on 25 inferentially complex circumstances from the National Violent Death Reporting System (NVDRS). We found that LLMs substantially outperform on low-prevalence circumstances where training data is insufficient. We further demonstrate that our framework generalizes across frontier LLMs, with GPT-5.2, Gemini 2.5 Pro and Llama-3 70B showing consistent performance patterns. These findings support a hybrid architecture where LLMs handle rare, inferentially complex circumstances while fine-tuned models handle common ones.
