Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper outlines a reproducible pipeline for training a supervised classifier specifically tailored for pathology report classification, addressing the significant out-of-distribution performance drop when models are applied across different cancer registries. The methodology includes facility-stratified sampling and a blinded manual audit to improve data quality and reduce label noise, resulting in a classifier that achieves a false-negative rate of 0.003 and an F1 score of 0.922 on a large holdout set. These results demonstrate a substantial improvement over existing models, highlighting the importance of in-domain training for reliable biomedical NLP applications.
Achieving a false-negative rate of just 0.003, this pipeline significantly enhances the reliability of pathology report classification across diverse cancer registries.
We introduce an in-domain supervised pipeline designed to counter the out-of-distribution performance drop that hampers supervised biomedical NLP models, a problem observed when models trained on pathology reports are moved across cancer registries. Our contribution is a reproducible recipe for training a supervised classifier from routinely collected cancer registry data. It describes how to build the in-domain training set and a production-matched holdout, and to choose operating points that keep the false-negative rate (FNR) very low while keeping reviewer workload manageable. The pipeline standardizes data curation with facility-stratified sampling and separate handling of reports linked to registry cases, and includes a blinded manual audit to estimate positive-case prevalence and label noise. On a 418k-report holdout set, the Kentucky model achieved FNR 0.003 and false-positive rate (FPR) 0.097, improving over the Seattle-trained MOSSAIC OncoID baseline (FNR 0.010, FPR 0.183) and raising F1 from 0.860 to 0.922. In a blinded manual review of 600 reports, estimated positive prevalence declined from 0.500 to 0.398, indicating substantial label noise with errors concentrated in rare primary sites.