Search papers, labs, and topics across Lattice.
This paper introduces ASAP, an innovative agent-system co-design for hyperparameter optimization (HPO) that leverages large language models (LLMs) to integrate a diverse range of inductive-biased optimizers while optimizing for wall-clock time. By addressing the limitations of existing HPO methods, which either rely on a single inductive bias or fail to account for real-world execution times, ASAP enhances sample efficiency and reduces overall runtime. Extensive experiments demonstrate that ASAP significantly outperforms traditional baselines across various modern HPO tasks, highlighting the effectiveness of its integrated approach.
ASAP achieves faster and more efficient hyperparameter optimization by integrating diverse optimizers and re-architecting the evaluation loop, leading to substantial wall-clock time reductions.
Hyperparameter Optimization (HPO) is essential for maximizing machine learning model performance, and its core challenge is sample efficiency: finding strong configurations within a limited budget. Because every HPO tool relies on a surrogate prior that imparts its own inductive bias, individual tools struggle once problems become sufficiently diverse and drift from these priors. Motivated by the reasoning and generalization capabilities of LLMs, recent work has explored using LLMs for HPO and reports improved per-iteration performance. Yet these methods share two limitations with a common origin: they use the LLM as a single-tool replacement evaluated by iteration count. (i) Deployed in place of prior tools, the LLM is itself constrained by its pretraining objective to one family of inductive-biased proposals; this single-source setup still fails to handle the full diversity of problems. (ii) Per-iteration evaluation ignores that, in real runs, LLM inference or tool execution is paid serially on top of model evaluation every round, so iteration-count gains do not translate into end-to-end wall-clock gains. We present ASAP, an agent-system co-design that addresses both limitations. On the agent side, ASAP uses the LLM to integrate a diverse pool of inductive-biased optimizers and to select among their proposals each round. On the system side, ASAP re-architects the loop to reduce end-to-end wall-clock while preserving regret quality: a prefix-stable prompt maximizes KV-cache reuse across rounds; speculation parallelism hides the remaining LLM and tool latency under model evaluation via a relative-error accept test; and a Self-Tuner adapts the speculation threshold from execution logs off the critical path. Extensive experiments on diverse modern HPO tasks show that ASAP consistently outperforms baselines, underscoring the value of tool integration and agent-system co-design.
