Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a pipeline for automatically generating challenging math problems tailored to expose weaknesses in LLMs. The pipeline uses AI-generated hypotheses to pinpoint specific mathematical concepts where LLMs falter, then crafts new benchmark problems targeting these identified vulnerabilities. Experiments demonstrate that problems generated from high-confidence hypotheses significantly reduce the accuracy of Llama-3.3-70B-Instruct, highlighting the effectiveness of the approach in creating targeted and difficult benchmarks.
LLMs that ace standard math benchmarks can still be stumped by problems automatically generated to target their specific weaknesses, revealing a crucial gap in current evaluation methods.
Numerous math benchmarks exist to evaluate LLMs' mathematical capabilities. However, most involve extensive manual effort and are difficult to scale. Consequently, they cannot keep pace with LLM development or easily provide new instances to mitigate overfitting. Some researchers have proposed automatic benchmark generation methods, but few focus on identifying the specific math concepts and skills on which LLMs are error-prone, and most can only generate category-specific benchmarks. To address these limitations, we propose a new math benchmark generation pipeline that uses AI-generated hypotheses to identify the specific math concepts and skills that LLMs struggle with, and then generates new benchmark problems targeting these weaknesses. Experiments show that hypothesis accuracy positively correlates with the difficulty of the generated problems: problems generated from the most accurate hypotheses reduce Llama-3.3-70B-Instruct's accuracy to as low as 45%, compared to 77% on the original MATH benchmark. Furthermore, our pipeline is highly adaptable and can be applied beyond math to explore a wide range of LLM capabilities, making it a valuable tool for investigating how LLMs perform across different domains.