Search papers, labs, and topics across Lattice.
This paper introduces SemanticAlign-Bench (SA-Bench), a diagnostic benchmark designed to evaluate the semantic alignment of LLM-generated code against the specifications of scientific papers. By decomposing paper specifications into Semantic Alignment Units (SAUs) and assessing them across four dimensions of drift, the authors reveal that even the best-performing model configuration achieves a mean SAU score of only 0.301, highlighting significant shortcomings in LLMs' ability to reproduce scientific implementations faithfully. The findings underscore the need for improved scaffolding that emphasizes semantic verification to enhance the reliability of LLM-generated code in scientific contexts.
LLMs struggle with semantic fidelity, achieving only a 30% alignment with scientific paper specifications, revealing a critical gap in AI-driven research reproducibility.
LLM agents can generate paper reproduction code, yet often produce scientifically unfaithful implementations. We define this failure mode as semantic drift, where generated code silently diverges from the paper's specifications. We introduce SemanticAlign-Bench(SA-Bench), a diagnostic benchmark covering 30 papers from ICLR, ICML and NeurIPS 2025. For each paper, we decompose its specifications into atomic and verifiable implementation claims, which we call Semantic Alignment Units (SAUs) and evaluate repositories along four diagnostic dimensions spanning numerical, methodological, protocol and ordering drift. In total, we construct 1,491 SAUs across five ML domains and evaluate 12 generator configurations (4 models $\times$ 3 scaffolds). Even the strongest configuration (Claude+PaperCoder) achieves a mean SAU score of only 0.301 out of 1.0, with an overall mean of 0.221 across 360 evaluations. A failure taxonomy reveals that agents attempt most requirements but implement them incorrectly, with implementation mismatch and stubs accounting for the majority of zero-scored claims. Our analysis further indicates that scaffolds optimized for executability provide limited leverage for scientific reproduction; narrowing the gap requires scaffolds that prioritize semantic specification verification. The benchmark, annotations and evaluation pipeline are publicly available.
