Search papers, labs, and topics across Lattice.
This paper introduces a framework to evaluate LLM decision-making in data-constrained scientific workflows, separating stability, correctness, prompt sensitivity, and output validity. Using a statistical gene prioritization task, the authors demonstrate that LLMs can exhibit high stability while systematically deviating from statistical ground truth. The study reveals failure modes such as over-selection under relaxed thresholds, sensitivity to minor prompt changes, and generation of invalid gene identifiers, highlighting the limitations of relying solely on stability for LLM validation.
LLMs can be confidently wrong in scientific decision-making, exhibiting near-perfect stability while systematically diverging from statistical ground truth.
Large language models (LLMs) are increasingly used as decision-support tools in data-constrained scientific workflows, where correctness and validity are critical. However, evaluation practices often emphasize stability or reproducibility across repeated runs. While these properties are desirable, stability alone does not guar- antee agreement with statistical ground truth when such references are available. We introduce a controlled behavioral evaluation framework that explicitly sep- arates four dimensions of LLM decision-making: stability, correctness, prompt sensitivity, and output validity under fixed statistical inputs. We evaluate multi- ple LLMs using a statistical gene prioritization task derived from differential ex- pression analysis across prompt regimes involving strict and relaxed significance thresholds, borderline ranking scenarios, and minor wording variations. Our ex- periments show that LLMs can exhibit near-perfect run-to-run stability while sys- tematically diverging from statistical ground truth, over-selecting under relaxed thresholds, responding sharply to minor prompt wording changes, or producing syntactically plausible gene identifiers absent from the input table. Although sta- bility reflects robustness across repeated runs, it does not guarantee agreement with statistical ground truth in structured scientific decision tasks. These findings highlight the importance of explicit ground-truth validation and output validity checks when deploying LLMs in automated or semi-automated scientific work- flows.
