Search papers, labs, and topics across Lattice.
The paper introduces Auto, a novel compiler that captures and optimizes the behavior of LLM agents by identifying deterministic components and converting them into verified programs, thereby reducing operational costs significantly. Through a rigorous evaluation on the AUTO-BENCH benchmark, Auto demonstrated that 87.1% of recorded agent behaviors are deterministically reproducible, achieving a remarkable cost reduction from 59 to 2 micro-dollars per processed item while maintaining high accuracy. The findings highlight that the fidelity of calibration and reference models, rather than sheer model size, is critical for ensuring low-cost, reliable performance in AI systems.
Capturing and optimizing LLM agent behavior can slash operational costs by over 90% while maintaining high accuracy, challenging assumptions about model capability.
Every LLM agent run re-derives its behavior token by token on a frontier model: brilliant, expensive, slow, and unbounded. We present Auto, a compiler that records live agent behavior, measures which parts are secretly deterministic, extracts them into verified programs or distilled specialists, and emits cognition binaries: WebAssembly artifacts whose manifests carry measured guarantees and whose declared capabilities are physically enforced by the sandbox. A tiered runtime executes compiled behavior behind conformally calibrated guards; guard trips deopt to the reference agent, and the captured trace recompiles back down, so nothing is figured out twice. We use "AGI compiler" in one narrow, testable sense: a system that autonomously converts novel experience into permanent, verified, near-free skill while measuring what it does not know. On AUTO-BENCH, a benchmark we introduce and pre-register, 87.1% of 560 recorded frontier-agent spans are witnessed-deterministic (three of the four censused task families measure 100.0%). On a 300-item stream with three scheduled distribution shifts, the closed loop compiles three artifact generations and drives marginal cost from 59 to 2 micro-dollars per item (6.4x end-to-end) at 96.9% parity on witnessed inputs with zero errors. The same stream also quantifies the failure modes: a loose guard silently mislabels 48.9% of compiled answers, and an unfaithful deopt reference causes the verification gate to refuse recompilation. Calibration and reference fidelity, not model capability, decide whether cheap stays correct. Code: https://github.com/RightNow-AI/auto
