Search papers, labs, and topics across Lattice.
This paper introduces a hybrid nested search framework that separates the structural and parameter optimization tasks in LLM-driven evolutionary algorithms, addressing inefficiencies in token usage during discrete updates. By employing a two-loop system where an LLM proposes structural sketches and a numerical optimizer fine-tunes them, the authors demonstrate significant improvements in optimization performance. Validation across three scientific domains shows that this approach outperforms both traditional LLM-driven searches and pure numerical optimization methods.
Hybrid nested optimization outperforms traditional methods by effectively decoupling structural design from parameter tuning in LLM-driven evolutionary algorithms.
In evolutionary algorithms powered by language models, the LLM acts as a single operator that simultaneously updates structural components (like control flow) and continuous parameters. While LLMs can be good at the first, they are not efficient at the second, wasting tokens taking discrete jumps inside a trial and error loop. We resolve this by formalizing a hybrid nested search, in which an outer loop has the LLM propose a structural sketch, with numeric gaps, and an inner numerical optimizer tunes the sketch. Both the outer and inner solvers are pluggable: any text-based optimizer can be combined with a zero-order optimizer (CMA-ES), gradient-based routines, or MCMC samplers. We validate our framework across three scientific domains: (i) meta-optimizers on closed-form test functions, (ii) code-based policies for systems research and social dilemmas; and (iii) approximate Bayesian inference tasks. Across all three, the hybrid optimizer is superior to both vanilla LLM-driven search and pure numerical optimization baselines. Code at: https://github.com/vicgalle/hybrid-nested-search
