Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores using GPT-4.1 to predict fan experience ratings (0-10 scale) based solely on open-ended text responses from baseball fans. The study compares these AI predictions against actual survey ratings across 10,000 responses, finding that two-thirds of predicted ratings were within one point of self-reported ratings. The analysis reveals that while predicted ratings correlate strongly with overall experience, they systematically underestimate the self-reported ratings, suggesting that LLMs capture salient moments while self-reports reflect a holistic "verdict."
GPT-4 can predict baseball fan experience ratings from text with surprising accuracy, but the systematic gap between AI predictions and self-reports reveals a fundamental difference between capturing salient moments and rendering a holistic judgment.
We tasked GPT-4.1 to read what baseball fans wrote about their game-day experience and predict the overall experience rating each fan gave on a 0-10 survey scale. The model received only the text of a single open-ended response. These AI predictions were compared with the actual experience ratings captured by the survey instrument across approximately 10,000 fan responses from five Major League Baseball teams. In total two-thirds of predicted ratings fell within one point of self-reported fan ratings (67% within +/-1, 36% exact match), and the predicted measurement was near-deterministic across three independent scoring runs (87% exact agreement, 99.9% within +/-1). Predicted ratings aligned most strongly with the overall experience rating (r = 0.82) rather than with any specific aspect of the game-day experience such as parking, concessions, staff, etc. However, predictions were systematically lower than self-reported ratings by approximately one point, and this gap was not driven by any single aspect. Rather, our analysis shows that self-reported ratings capture the fan's verdict, an overall evaluative judgment that integrates the entire experience. While predicted ratings quantify the impact of salient moments characterized as memorable, emotionally intense, unusual, or actionable. Each measure contains information the other misses. These baseline results establish that a simple, unoptimized prompt can directionally predict how fans rate their experience from the text a fan wrote and that a gap between the two numbers can be interpreted as a construct difference worth preserving rather than an error to eliminate.