Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
The authors construct BreCol, a benchmark assessing classical models and genomic deep learning architectures (HyenaDNA, SetBERT) across 2,040 16S rRNA sequencing runs spanning 26 studies to evaluate cross-study generalization in microbiome-based cancer detection. Rigorous out-of-distribution evaluation is critical in metagenomics, where technical batch effects and cohort discrepancies frequently inflate reported in-distribution diagnostic accuracy. Surprisingly, specialized deep sequence models underperform classical pipelines on temporally separated holdouts, where overall cancer diagnostic performance degrades significantly (AUC dropping from 0.77 to 0.60).
Genomic foundation models like HyenaDNA and SetBERT fail to beat simple k-mer clustering on out-of-distribution microbiome cohorts, exposing a severe generalization gap across clinical studies.
DNA sequencing of the gut microbial community shows promise for cancer detection, but questions remain about the generalizability of results across studies. We propose BreCol, a benchmark of 2,040 16S rRNA gene sequencing runs across 26 studies spanning breast cancer, colorectal cancer, and healthy cohorts. Train-test splits are made within pre-2023 studies, while holdout evaluation uses studies from 2023 onward, reflecting temporal separation from training data. Classical models reach test/holdout AUCs of 0.77/0.60 for cancer diagnosis and 1.00/0.83 for cancer type prediction. We train the models on both cancer types simultaneously and find that colorectal cancer is often easier to detect than breast cancer. We also evaluate two deep learning models: HyenaDNA, a long-range sequence model that pools hidden states for classification, and SetBERT, a transformer that produces contextualized embeddings over sets of reads. Both deep learning models underperform the best classical methods on holdout data, though tuning training set size and the classification head yields modest gains. Our classical pipeline uses unsupervised clustering to derive features from tetramer frequencies, preserving within-run compositional signal and achieving near state-of-the-art performance without relying on taxonomic assignments. BreCol data and associated code are publicly available.