Search papers, labs, and topics across Lattice.
TCS-Bench is a newly introduced benchmark designed to assess the ability of Large Language Models (LLMs) in generating proofs for research-level Theoretical Computer Science (TCS) problems. By utilizing theorem-proving tasks derived from prestigious conferences, the benchmark provides a structured context for evaluating LLMs' proof generation capabilities. The study finds that the reference verifier achieves over 90% accuracy in validating the proofs generated by these models against human expert judgments, highlighting the potential for LLMs in formal reasoning tasks.
LLMs can now tackle complex TCS proof generation with a benchmark that achieves over 90% accuracy in verification against human experts.
We introduce TCS-Bench, a benchmark for evaluating Large Language Models (LLMs) on research-level Theoretical Computer Science (TCS) proof generation. TCS-Bench consists of theorem-proving tasks from papers published at top theoretical computer science venues (STOC, FOCS, and SODA). Each task provides the necessary context to derive a self-contained proof for a target result. We evaluate state-of-the-art models on this benchmark. We verify the correctness of generated proofs via a verification agent, and further benchmark the verifier against human-expert proof judgements on a set of target statements and generated proofs pairs. Our reference verifier achieves over 90% accuracy on the expert labeled set.
