Search papers, labs, and topics across Lattice.
This study investigates the recovery of scientifically meaningful mechanisms in AI-generated materials-science hypotheses using a graph-to-answer tracing approach with the Graph-PRefLexOR-8B model. By employing a visual diagnostic workflow that incorporates semantic backtracking and activation-based recovery measurements, the authors analyze the model's performance across 100 open-ended questions. The findings reveal that while synthesis stages maintain closer alignment with structured reasoning, significant mechanism recovery is notably absent in earlier layers, highlighting critical points where hypotheses may lack scientific validity.
Mechanism recovery in AI-generated hypotheses falters in early reasoning stages, revealing vulnerabilities in scientific validity that could mislead experimental planning.
AI co-scientists can generate fluent materials-science hypotheses, but fluency does not show that an answer preserves a scientifically meaningful mechanism. We present a graph-to-answer mechanism-tracing case study for Graph-PRefLexOR-8B, a Qwen3-8B model adapted to expose distinct stages for brainstorming, graph construction, pattern extraction, and synthesis. We organize semantic backtracking, graph corruption, activation-based recovery measurements, and layer-by-token-region grids into a visual diagnostic workflow for inspecting this pathway. Across 100 open-ended materials-science questions, final answers remain closest to the model's own structured stages, especially synthesis. Under graph corruption, a full sweep over 37 residual-stream checkpoints, the embedding output and 36 transformer blocks, shows little mechanism recovery in the earlier transition region at layers 7--10, recovery instead concentrates in late synthesis and answer-start regions around layers 30 and 36. The workflow is intended to help scientists and model developers identify where a generated hypothesis loses or regains mechanism support before it is passed to downstream experimental planning.
