Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We introduce SOL, a novel metric for comparing text distributions by treating sequences as empirical measures of their transformer hidden states and comparing these measures using the double sliced Wasserstein distance. This approach addresses the limitations of perplexity and existing sample-based metrics for non-autoregressive models by directly assessing distributional fit. Experiments demonstrate SOL's ability to identify distributional discrepancies, align with expected model behavior, and offer stable estimations, filling a critical gap in current text generation evaluation.
Text generation evaluation can now directly measure distributional divergence using hidden states and double sliced Wasserstein distance, bypassing perplexity's limitations for non-autoregressive models.
Evaluating text generation requires measuring how well the generated distribution matches the data distribution. For autoregressive models, this is done by the perplexity. Diffusion and flow-based language models can only provide a likelihood bound, whose tightness differs between model families. Sample-based substitutes such as generative perplexity with entropy do not consider the distribution fit. We propose SOL, a distance between text distributions. Each sequence is represented by the empirical measure of its hidden states under a fixed transformer and the distributions of these measures are compared by the double sliced Wasserstein distance. We prove that SOL is a metric if the transformer is injective. Experiments show that SOL detects distributional failures, recovers expected model trends, and provides stable sample-based estimates. We put forward SOL to fill the gap in the current evaluation protocol used for non auto-regressive models. As a first step we use SOL to re-evaluate a variety of models trained on OpenWebText.