Search papers, labs, and topics across Lattice.
This study introduces the URSA evaluation framework, which benchmarks retrosynthesis pathways by assessing both formal convergence to commercially available starting materials and chemical plausibility, akin to expert chemist evaluations. By applying this framework to a diverse set of target molecules with undisclosed synthetic routes, the authors comprehensively evaluate both conventional retrosynthesis models and large language models (LLMs). The findings reveal that while LLMs excel in high-level strategic planning, they fall short compared to specialized models in reliably executing synthesis planning tasks.
LLMs may excel at strategic thinking, but they lag behind specialized models in practical synthesis planning, highlighting a critical gap in current AI capabilities.
Synthesis planning aiming to find pathways of reactions for a target molecule is one of the most important and challenging tasks in drug discovery. Recent progress has produced both specialized deep-learning retrosynthesis systems and general-purpose large language models, but objective comparison remains difficult due to the lack of flexible, chemically interpretable benchmarking protocols. In the current study, we are introducing the URSA (Utilitarian RetroSynthesis Assessment) evaluation framework that provides the opportunity to benchmark the synthetic routes not only from a formal perspective, such as convergence to commercially available starting materials, but also from a chemical plausibility perspective, mimicking the way expert chemists evaluate the reactions and routes. The study covers a comprehensive evaluation of both conventional end-to-end retrosynthesis solutions and LLMs for the synthesis planning task on a set of novel, diverse target molecules with undisclosed synthetic routes, which represent realistic tasks in the daily drug design routine. We find that while LLMs can support high-level strategic planning, they currently underperform specialized retrosynthesis models in reliably solving synthesis planning tasks.
