Search papers, labs, and topics across Lattice.
This paper investigates the prevalent but informal practice of "vibe-testing" LLMs, where users evaluate models based on personalized experiences rather than standardized benchmarks. Through a survey and analysis of real-world model comparisons, the authors formalize vibe-testing as a process involving personalized prompts and user-aware subjective criteria. They then introduce a proof-of-concept evaluation pipeline demonstrating that this formalized approach can alter model preferences compared to standard benchmarks, highlighting the importance of personalized evaluation.
Standard LLM benchmarks miss the mark: personalized "vibe-testing" reveals that user-specific prompts and subjective criteria can flip model rankings.
Evaluating LLMs is challenging, as benchmark scores often fail to capture models'real-world usefulness. Instead, users often rely on ``vibe-testing'': informal experience-based evaluation, such as comparing models on coding tasks related to their own workflow. While prevalent, vibe-testing is often too ad hoc and unstructured to analyze or reproduce at scale. In this work, we study how vibe-testing works in practice and then formalize it to support systematic analysis. We first analyze two empirical resources: (1) a survey of user evaluation practices, and (2) a collection of in-the-wild model comparison reports from blogs and social media. Based on these resources, we formalize vibe-testing as a two-part process: users personalize both what they test and how they judge responses. We then introduce a proof-of-concept evaluation pipeline that follows this formulation by generating personalized prompts and comparing model outputs using user-aware subjective criteria. In experiments on coding benchmarks, we find that combining personalized prompts and user-aware evaluation can change which model is preferred, reflecting the role of vibe-testing in practice. These findings suggest that formalized vibe-testing can serve as a useful approach for bridging benchmark scores and real-world experience.
