Search papers, labs, and topics across Lattice.
This paper investigates how LLMs assess the scientific feasibility of hypotheses, testing the impact of providing experimental descriptions versus outcome data as context. The authors find that providing outcome evidence generally leads to more reliable feasibility assessments compared to providing experimental descriptions. Furthermore, they show that experimental text can degrade performance when the context is incomplete, highlighting the fragility of LLMs' reliance on experimental details.
LLMs are better at judging scientific feasibility when given results, not methods, suggesting they struggle to simulate experiments.
Scientific feasibility assessment asks whether a claim is consistent with established knowledge and whether experimental evidence could support or refute it. We frame feasibility assessment as a diagnostic reasoning task in which, given a hypothesis, a model predicts feasible or infeasible and justifies its decision. We evaluate large language models (LLMs) under controlled knowledge conditions (hypothesis-only, with experiments, with outcomes, or both) and probe robustness by progressively removing portions of the experimental and/or outcome context. Across multiple LLMs and two datasets, providing outcome evidence is generally more reliable than providing experiment descriptions. Outcomes tend to improve accuracy beyond what internal knowledge alone provides, whereas experimental text can be brittle and may degrade performance when the context is incomplete. These findings clarify when experimental evidence benefits LLM-based feasibility assessment and when it introduces fragility.
