Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper investigates the statistical independence assumption in turn-level analysis of LLM conversations, revealing significant autocorrelation between consecutive turns. Through a systematic analysis of 66 metrics across 202 conversations, the authors demonstrate that ignoring this autocorrelation leads to a 42% inflation of spurious significance in standard pooled testing. They propose and validate a two-stage correction framework using effective degrees of freedom and conversation-level block bootstrap, showing improved replication rates on a hold-out set.
Nearly half of published findings about LLM behavior in conversations may be statistically invalid due to a widespread failure to account for autocorrelation between turns.
Turn-level metrics are widely used to evaluate properties of multi-turn human-LLM conversations, from safety and sycophancy to dialogue quality. However, consecutive turns within a conversation are not statistically independent -- a fact that virtually all current evaluation pipelines fail to correct for in their statistical inference. We systematically characterize the autocorrelation structure of 66 turn-level metrics across 202 multi-turn conversations (11,639 turn pairs, 5 German-speaking users, 4 LLM platforms) and demonstrate that naive pooled analysis produces severely inflated significance estimates: 42% of associations that appear significant under standard pooled testing fail to survive cluster-robust correction. The inflation varies substantially across categories rather than scaling linearly with autocorrelation: three memoryless families (embedding velocity, directional, differential) aggregate to 14%, while the seven non-memoryless families (thermo-cycle, frame distance, lexical/structural, rolling windows, cumulative, interaction, timestamp) aggregate to 33%, with individual category rates ranging from 0% to 100% depending on per-family effect size. We present a two-stage correction framework combining Chelton (1983) effective degrees of freedom with conversation-level block bootstrap, and validate it on a pre-registered hold-out split where cluster-robust metrics replicate at 57% versus 30% for pooled-only metrics. We provide concrete design principles, a publication checklist, and open-source code for the correction pipeline. A survey of ~30 recent papers at major NLP and AI venues that compute turn-level statistics in LLM evaluations finds that only 4 address temporal dependence at all, and 26 do not correct for it.