Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of five open-source generative Large Language Models (LLMs) in a zero-shot workflow for extracting critical data from lung pathology reports, addressing the limitations of traditional supervised methods that require extensive manual annotation. The best-performing model, GPT-OSS-20B, achieved a Micro-F1 score of 0.893, demonstrating strong recall and the ability to accurately extract complex relationships without task-specific training. These findings highlight the potential of zero-shot LLMs as a cost-effective alternative for information extraction in clinical settings, particularly for cancer staging and tumor registry population.
Open-source, zero-shot LLMs can extract lung pathology information with impressive accuracy, offering a low-cost alternative to labor-intensive manual processes.
Information extraction from pathology reports is essential for cancer staging, tumor registry population. Yet key data remains embedded in narrative reports, making manual extraction labor-intensive and error-prone. Traditional supervised Natural Language Processing pipelines address this through fully supervised Named Entity Recognition and Relation Extraction, but require expensive manual annotation and suffer cascading failures when upstream entities are missed. In this study, we developed a zero-shot, agentic workflow, and evaluated five open-source generative Large Language Models (LLMs) to populate 13 College of American Pathologists synoptic fields from lung resection pathology reports. We compared them against a state-of-the-art supervised GatorTron NER-RE baseline using a novel, registry-aligned evaluation framework. The baseline achieved Micro-F1of 0.960, while the best zero-shot model (GPT-OSS-20B) achieved Micro-F1 of 0.893 (recall: 0.949), accurately extracting complex relations like Pathologic Stage without task-specific training. These results suggest that open-source, zero-shot agentic LLMs are a low-cost solution for extracting lung pathology information.
