Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to evaluating generative models by analyzing the distribution of discrete visual tokens produced by 1D image tokenizers, arguing that these tokens preserve perceptual cues lost in continuous feature-based metrics like FID. They propose two new metrics: Codebook Histogram Distance (CHD), a training-free distribution metric, and Code Mixture Model Score (CMMS), a learned no-reference quality metric. Evaluated on a new benchmark, VisForm, and existing datasets, their token-based metrics demonstrate state-of-the-art correlation with human perceptual judgments.
Ditch blurry FID scores: evaluating generative models directly in discrete token space reveals perceptual quality nuances that continuous feature metrics miss.
Most evaluations of generative models rely on feature-distribution metrics such as FID, which operate on continuous recognition features that are explicitly trained to be invariant to appearance variations, and thus discard cues critical for perceptual quality. We instead evaluate models in the space of \emph{discrete} visual tokens, where modern 1D image tokenizers compactly encode both semantic and perceptual information and quality manifests as predictable token statistics. We introduce \emph{Codebook Histogram Distance} (CHD), a training-free distribution metric in token space, and \emph{Code Mixture Model Score} (CMMS), a no-reference quality metric learned from synthetic degradations of token sequences. To stress-test metrics under broad distribution shifts, we further propose \emph{VisForm}, a benchmark of 210K images spanning 62 visual forms and 12 generative models with expert annotations. Across AGIQA, HPDv2/3, and VisForm, our token-based metrics achieve state-of-the-art correlation with human judgments, and we will release all code and datasets to facilitate future research.
