Search papers, labs, and topics across Lattice.
The paper introduces Distribution Map (DMAP), a novel method for representing text using next-token probability distributions from LLMs by mapping text to samples in the unit interval that encode rank and probability. DMAP addresses the limitations of perplexity by accounting for context and the shape of the conditional distribution. The authors demonstrate DMAP's utility in validating generation parameters, detecting machine-generated text via probability curvature, and performing forensic analysis of models fine-tuned on synthetic data.
Forget perplexity: DMAP offers a mathematically grounded, model-agnostic representation of text that unlocks new insights into generation quality, machine-generated text detection, and forensic analysis of synthetic data influence.
Large Language Models (LLMs) are a powerful tool for statistical text analysis, with derived sequences of next-token probability distributions offering a wealth of information. Extracting this signal typically relies on metrics such as perplexity, which do not adequately account for context; how one should interpret a given next-token probability is dependent on the number of reasonable choices encoded by the shape of the conditional distribution. In this work, we present DMAP, a mathematically grounded method that maps a text, via a language model, to a set of samples in the unit interval that jointly encode rank and probability information. This representation enables efficient, model-agnostic analysis and supports a range of applications. We illustrate its utility through three case studies: (i) validation of generation parameters to ensure data integrity, (ii) examining the role of probability curvature in machine-generated text detection, and (iii) a forensic analysis revealing statistical fingerprints left in downstream models that have been subject to post-training on synthetic data. Our results demonstrate that DMAP offers a unified statistical view of text that is simple to compute on consumer hardware, widely applicable, and provides a foundation for further research into text analysis with LLMs.
