Search papers, labs, and topics across Lattice.
This paper introduces a unified framework for ensembling language models into $f$-ensemble distributions, addressing the challenge of biased approximations when naively aggregating next-token probabilities. They propose a byte-level sequential Monte Carlo (SMC) algorithm to sample from these distributions, enabling ensembles of models with mismatching vocabularies. Experiments on structured text generation tasks demonstrate the benefits of alternative aggregation strategies over probability averaging, showing that better posterior approximations lead to improved ensemble performance.
Forget simple probability averaging: a byte-level sequential Monte Carlo algorithm unlocks better language model ensembles by enabling sophisticated aggregation strategies and handling mismatched vocabularies.
Practitioners have access to an abundance of language models and prompting strategies for solving many language modeling tasks; yet prior work shows that modeling performance is highly sensitive to both choices. Classical machine learning ensembling techniques offer a principled approach: aggregate predictions from multiple sources to achieve better performance than any single one. However, applying ensembling to language models during decoding is challenging: naively aggregating next-token probabilities yields samples from a locally normalized, biased approximation of the generally intractable ensemble distribution over strings. In this work, we introduce a unified framework for composing $K$ language models into $f$-ensemble distributions for a wide range of functions $f\colon\mathbb{R}_{\geq 0}^{K}\to\mathbb{R}_{\geq 0}$. To sample from these distributions, we propose a byte-level sequential Monte Carlo (SMC) algorithm that operates in a shared character space, enabling ensembles of models with mismatching vocabularies and consistent sampling in the limit. We evaluate a family of $f$-ensembles across prompt and model combinations for various structured text generation tasks, highlighting the benefits of alternative aggregation strategies over traditional probability averaging, and showing that better posterior approximations can yield better ensemble performance.
