Search papers, labs, and topics across Lattice.
The paper introduces "diversity coverage," a metric to evaluate how well LLMs generate diverse and high-quality responses to open-ended prompts. Evaluating 18 LLMs reveals that no single model consistently excels at generating diverse responses across various prompts, but for each prompt, a specific model often significantly outperforms others. To exploit this, the authors train a router to predict the best model for each query, achieving improved diversity coverage compared to using a single best model.
Turns out, ensembling LLMs for diversity isn't about picking the biggest model, but routing each prompt to the *right* model.
When posed with prompts that permit a large number of valid answers, comprehensively generating them is the first step towards satisfying a wide range of users. In this paper, we study methods to elicit a comprehensive set of valid responses. To evaluate this, we introduce \textbf{diversity coverage}, a metric that measures the total quality scores assigned to each \textbf{unique} answer in the predicted answer set relative to the best possible answer set with the same number of answers. Using this metric, we evaluate 18 LLMs, finding no single model dominates at generating diverse responses to a wide range of open-ended prompts. Yet, per each prompt, there exists a model that outperforms all other models significantly at generating a diverse answer set. Motivated by this finding, we introduce a router that predicts the best model for each query. On NB-Wildchat, our trained router outperforms the single best model baseline (26.3% vs $23.8%). We further show generalization to an out-of-domain dataset (NB-Curated) as well as different answer-generation prompting strategies. Our work lays foundation for studying generating comprehensive answers when we have access to a suite of models.
