Search papers, labs, and topics across Lattice.
This paper introduces COMPAS, a difficulty-aware optimization method for code generation that jointly tunes models, prompts, and decoding settings based on task difficulty. By learning group-specific quality-cost fronts, COMPAS routes tasks to the optimal configuration without additional search, resulting in a significant improvement in pass rates and cost efficiency. The method achieved a pass@1 score of 52.8% on LiveCodeBench, outperforming the best baseline while dramatically reducing operational costs.
COMPAS boosts code generation performance by 15% while slashing costs by over 86%, revealing the critical interplay between task difficulty and optimization choices.
Code generation systems make each LLM call with a model, a prompt, and decoding settings. However, existing optimization methods usually tune only part of these choices or use one fixed configuration for all tasks: global optimizers search one configuration for all tasks, routers choose only a model, and prompt optimizers keep the model and decoding settings fixed. This leaves their joint, group-specific interactions unclear. We therefore examine how these choices interact and observe that prompts and decoding settings interact, tuning effects vary by model, and the best configuration varies by task difficulty. Guided by these observations, we introduce COMPAS (Code-generation Optimization over Models, Prompts, And Decoding Settings), a difficulty-aware method that learns group-specific quality-cost fronts through low-cost model selection and joint prompt-decoding search, then routes each test task to its matching front online without further search. Under a matched search budget on LiveCodeBench, COMPAS improves pass@1 from 45.9% for the best baseline to 52.8% while reducing cost from $36.57 to $4.92. This also transfers to repository-level code generation on SWE-bench, resolving 76.0% of tasks versus 70.0% for the best baseline. Code and the reproducibility artifact are available at https://github.com/gjz78910/COMPAS.
