Search papers, labs, and topics across Lattice.
This paper introduces a multi-agent framework called quantity grounded semantic differencing, designed to enhance the transparency and reliability of AI-generated scientific analyses by decomposing tasks into separate agents for code generation, execution, tracing, and validation. By addressing uncertainties in both natural language specifications and generated implementations, the framework allows for a clearer understanding of the assumptions and computations involved in scientific results. Validation on collider physics analyses shows that this modular approach not only improves interpretability but also enables smaller models to effectively execute the entire workflow, outperforming traditional single prompt methods.
Modular task decomposition in AI-generated analyses boosts transparency and reliability, enabling smaller models to outperform larger counterparts.
Scientific results produced by LLM generated analysis code must be understandable and reproducible. However, uncertainty can arise at different stages of the process, both in the original natural language specification and in the generated implementation. As a result, even executable code may not provide a clear understanding of which quantities are being computed or which assumptions determine the final results. To address this challenge, we introduce quantity grounded semantic differencing, a multi-agent framework for analyzing and comparing scientific programs generated by LLMs. The framework assigns code generation, execution, tracing, and validation to separate agents, allowing it to reconstruct how key output quantities are produced and to identify differences between the intended analysis and the implemented code. We also introduce a module that inspects ambiguities in the initial user instruction and suggests alternative rewrites before code generation. Its modular design enables application to different scientific domains by replacing domain specific resources while preserving the same workflow. We validate the framework on representative collider physics analyses. The results demonstrate that the modular task decomposition enhances both transparency and reliability relative to the previous single prompt approach, while enabling substantially smaller models to execute the complete workflow.
