Search papers, labs, and topics across Lattice.
This study introduces a statistical-field framework to analyze text generated by large language models (LLMs), conceptualizing token embeddings as continuous spin variables along a one-dimensional chain. By varying the softmax temperature \( T \), the authors identify a sharp peak in susceptibility and a rapid change in the order parameter near a critical temperature \( T_c \), indicating a collapse into a single semantic direction below this threshold. The findings, which are consistent across various model scales and prompt categories, suggest a connection between decoding strategies and critical phenomena, although the non-equilibrium nature of autoregressive generation necessitates further exploration.
A critical temperature \( T_c \) reveals a phase transition-like behavior in LLM outputs, reshaping our understanding of how temperature scaling affects semantic coherence.
We propose a statistical-field framework for text generated by large language models (LLMs), treating token embeddings as continuous spin variables on a one-dimensional chain. Defining a susceptibility from the connected two-point correlator and an order parameter from the ensemble-averaged embedding field, we vary the \texttt{softmax} temperature $T$ and observe a sharp susceptibility peak near a characteristic $T_c$ with power-law-like scaling, a concurrent rapid change in the order parameter, and a collapse onto a single semantic direction below $T_c$. The intrinsic dimension estimated by the two nearest neighbor (TwoNN) method independently corroborates these findings, reaching a minimum near $T_c$. Results are robust across model scales (Qwen3: 0.6B--32B) and prompt categories. While the phenomenology closely resembles a continuous phase transition, the non-equilibrium nature of autoregressive generation warrants further investigation. Our framework provides quantitative tools for probing the collective statistical structure of LLM outputs and suggests connections between decoding strategies and critical phenomena.
