Search papers, labs, and topics across Lattice.
This paper introduces program distillation as a scalable alternative to LLMs for automated evaluation, addressing issues of cost, latency, and decision opacity. By synthesizing a committee of programs that mimic LLM decision logic, the proposed PAJAMA system achieves comparable performance to a 13B-size LLM judge while significantly reducing operational costs and improving throughput. Additionally, the programmatic judges provide transparent and editable decision-making processes, enhancing the overall reliability of evaluations and reward signal generation.
Programmatic judges can match the performance of large LLMs while slashing costs and boosting evaluation speed by orders of magnitude.
LLM-as-a-judge has become the standard for automated evaluation, but it suffers from high cost, significant latency, and opaque decisions -- limitations that undermine its scalability and reliability. We address these with a simple, efficient alternative: program distillation. Instead of prompting an LLM at the evaluation time, we distill its decision logic into a committee of programs that score candidates directly. These programmatic judges offer transparency, are easily inspected or edited, and eliminate per-sample API costs. Building on this notion, we introduce PAJAMA, a system that synthesizes programs as judges, aggregates their decisions into a joint verdict, and incorporates a fallback mechanism to selectively escalate low-confidence cases to an LLM. Across five datasets and four model families, we show that programmatic judges can match the performance of a 13B-size LLM judge. When using program outputs as routing signals, PAJAMA improves both accuracy and throughput and advances the Pareto frontier. Beyond evaluation, programmatic judges produce cheap and effective reward signals: on RewardBench, a reward model distilled from programs'verdicts outperforms one trained on a proprietary LLM's labels at two orders of magnitude lower API cost.
