Search papers, labs, and topics across Lattice.
The paper introduces A$^{2}$utoLPBench, an innovative benchmark for evaluating LLM-driven agents on linear programming (LP) problems generated through an inverse-KKT construction method. This approach allows for the creation of an unlimited supply of dynamic LP problems with known optimal solutions, circumventing the limitations of static, hand-labeled datasets. Key results demonstrate that the benchmark provides calibrated scores for agents while ensuring resistance to training data leakage and maintaining consistent problem difficulty through adjustable parameters.
A$^{2}$utoLPBench offers an endless stream of dynamically generated LP problems, ensuring agents can be tested against fresh challenges without the risk of training data leakage.
Most LP-from-text benchmarks are static datasets of word problems written and labeled by hand. Once such a dataset is released, its size is fixed, its difficulty is fixed, and every problem can leak into the training data of future LLMs. We present \textbf{A$^{2}$utoLPBench}, a benchmark for testing LLM-driven agents on linear programming problems written in plain text. We first pick a feasible point and dual, then write down a problem for which that point is optimal and the objective value is known. The answer is known by construction, with no solver call and no human annotator. The evaluation environment bundles a reference solver-critic baseline and a Docker image whose usage instructions are written for an LLM-driven agent to read. With these in place, any agent can run the benchmark and get a calibrated score with one command. Because the benchmark is a generator rather than a fixed dataset, it has properties no fixed dataset can match: an unlimited supply of fresh problems, a difficulty knob set by $(n,m)$, ground-truth answers correct by construction, low LLM-side cost per problem relative to human authoring, repeatable scores across independent batches, and resistance to training-data leakage when fresh post-cutoff seed ranges are used.
