Search papers, labs, and topics across Lattice.
Decoding-time policy composition for LLMs, which combines reward-specific policies via a weighted product of response probabilities, is prone to sampling bias when naively applied to next-token probabilities. This paper rigorously proves that an iterative correction based on Independence Metropolis-Hastings (MH) uniformly dominates Sampling-Importance-Resampling (SIR) across all convex f-divergences for a fixed rollout budget. Experiments in enumerable and LLM-scale settings validate these theoretical findings, demonstrating MH's superior performance in approximating the target distribution.
Metropolis-Hastings sampling for LLM policy composition consistently outperforms SIR, offering a provably better approximation of desired reward trade-offs with the same computational budget.
Post-training a large language model (LLM) often requires exploring trade-offs between multiple rewards, but retraining for each trade-off is expensive. Decoding-time policy composition allows these trade-offs to be adjusted by combining reward-specific policies at inference time. This composition targets a weighted product of the policies'probabilities over complete responses, but standard implementations combine their next-token probabilities, generally introducing sampling bias. We analyze a known iterative correction based on independence Metropolis-Hastings (MH). Our main result shows that, for every rollout budget, MH produces an output distribution at least as close to the target as sampling-importance-resampling (SIR) with the same budget, as measured by every convex f-divergence. We also derive a lower bound on MH's improvement over the uncorrected decoder in a consensus objective measuring agreement with the supplied policies. We further characterize the correction's sampling error in two asymptotic regimes: when the reward-specific policies approach agreement, and when the log ratio between target and uncorrected-decoder probabilities fluctuates increasingly widely, as can happen for long responses. We complement our analysis with experiments in enumerable and LLM-scale settings.
