Search papers, labs, and topics across Lattice.
This paper investigates the limitations of aggregate LLM benchmarks in capturing individual user preferences. It computes personalized model rankings using ELO ratings and Bradley-Terry coefficients for 115 Chatbot Arena users, revealing significant divergence from aggregate rankings. The study then correlates user query characteristics (topics and writing style) with LLM ranking variations, demonstrating that user heterogeneity substantially influences model preferences and can be used to predict user-specific rankings.
Aggregate LLM benchmarks mislead on individual preferences: model rankings correlate near-zero for over half of users.
With the rise in capabilities of large language models (LLMs) and their deployment in real-world tasks, evaluating LLM alignment with human preferences has become an important challenge. Current benchmarks average preferences across all users to compute aggregate ratings, overlooking individual user preferences when establishing model rankings. Since users have varying preferences in different contexts, we call for personalized LLM benchmarks that rank models according to individual needs. We compute personalized model rankings using ELO ratings and Bradley-Terry coefficients for 115 active Chatbot Arena users and analyze how user query characteristics (topics and writing style) relate to LLM ranking variations. We demonstrate that individual rankings of LLM models diverge dramatically from aggregate LLM rankings, with Bradley-Terry correlations averaging only $\rho = 0.04$ (57\% of users show near-zero or negative correlation) and ELO ratings showing moderate correlation ($\rho = 0.43$). Through topic modeling and style analysis, we find users exhibit substantial heterogeneity in topical interests and communication styles, influencing their model preferences. We further show that a compact combination of topic and style features provides a useful feature space for predicting user-specific model rankings. Our results provide strong quantitative evidence that aggregate benchmarks fail to capture individual preferences for most users, and highlight the importance of developing personalized benchmarks that rank LLM models according to individual user preferences.
