Search papers, labs, and topics across Lattice.
This paper introduces the Paper-replication workflow, which enables a coding agent to systematically replicate computational claims from scientific machine learning papers. By structuring the replication process around specific paper claims and requiring evidence recording and validation checks, the method ensures that the generated outputs are rigorously linked to the original claims. The evaluation of Paper-replication across twelve independent runs demonstrates that it successfully matches all recorded targets with report coverage, highlighting variability in execution while maintaining fidelity to the source papers.
Coding agents can now reliably replicate scientific claims, ensuring that computational results are not just generated but thoroughly validated against original research.
Scientific machine learning papers typically make computational claims, e.g., that the relative mean square error is less than 5% or that the 95% predictive credible interval covers the test data. A coding agent can be prompted to replicate those claims from paper materials alone, but the prompt does not by itself reliably preserve progress or check whether generated evidence supports the paper's claims. We introduce Paper-replication, a workflow that makes each selected paper claim a target with recorded evidence, and implement it as a coding-agent skill. The workflow makes the agent record those targets, reconstruct the paper's method, run computational experiments, link generated outputs to provenance and comparisons with the paper's claims, record where matched evidence appears in the replication report, and pass validation checks before completion. We evaluate Paper-replication on twelve independent runs across four scientific machine learning papers. All twelve workspaces pass the completion gate, and all 158 recorded targets are matched with report coverage. Even in this completed workspace state, repeated runs differ in how papers are divided into targets, in numerical fidelity to the source papers, in elapsed replication time, in the number of intermediate executions replaced before final evidence is accepted, and in the rules used to accept evidence. Paper-replication makes completion depend on workspace evidence and validation checks rather than on the agent's final message.
