Search papers, labs, and topics across Lattice.
This paper introduces \ourbenchmark, a new expert-validated benchmark designed to evaluate large language models (LLMs) on end-to-end theoretical computer science (TCS) research tasks. The benchmark consists of 175 instances derived from recent conference papers, maintaining the integrity of definitions, assumptions, and proof dependencies, and reveals that current LLMs struggle significantly with autoformalization, achieving only 11.5 in translating natural-language claims into formal statements. Furthermore, an automated TCS research framework was developed, which highlighted that only 6 out of 64 generated claims passed expert evaluation, underscoring the challenges of both formalization and research taste in autonomous TCS research.
Current LLMs can barely translate natural-language claims into formal statements, achieving only 11.5 on a critical task that could unlock automated theoretical research.
Large language models (LLMs) have shown growing potential for automated theoretical computer science (TCS) research, yet existing benchmarks remain far from realistic research settings. We introduce \ourbenchmark, an expert-validated benchmark for evaluating LLMs on frontier, end-to-end TCS research. \ourbenchmark contains $175$ instances drawn from papers accepted to STOC, FOCS, SODA, and COLT in 2025-2026, preserving paper-specific definitions, assumptions, and proof dependencies, with expert-verified Lean formalizations and proofs. Evaluations of leading LLMs reveal that current models remain far from reliably completing the full research pipeline. In particular, autoformalization is the sharpest bottleneck: the best model achieves only $11.5$ on translating natural-language claims into formal theorem statements, compared with $28.6$ Pass@8 when proving human-provided formal statements. Building on \ourbenchmark, we further develop an automated TCS research framework that generates, formalizes, filters, and proves new claims. Of $64$ generated claims, only $6$ ultimately pass expert evaluation and proof verification, indicating that beyond formalization, limited research taste remains another major barrier to autonomous TCS research.
