Search papers, labs, and topics across Lattice.
This paper introduces "model maps," a technique for representing language models as vectors of log-likelihoods over prompt-response pairs, enabling comparison of conditional distributions via distance metrics approximating KL divergence. Experiments on a diverse set of models reveal that these maps capture meaningful relationships between model attributes, task performance, and prompt modifications, including the approximate additive effects of composite prompts. The authors also explore pointwise mutual information (PMI) vectors to mitigate the influence of unconditional distributions, improving the reflection of training-data-related differences in some cases.
Forget comparing models with benchmarks – mapping them by prompt-response likelihoods reveals hidden relationships between architecture, training data, and even how prompts compose.
We propose a method that represents language models by log-likelihood vectors over prompt-response pairs and constructs model maps for comparing their conditional distributions. In this space, distances between models approximate the KL divergence between the corresponding conditional distributions. Experiments on a large collection of publicly available language models show that the maps capture meaningful global structure, including relationships to model attributes and task performance. The method also captures systematic shifts induced by prompt modifications and their approximate additive compositionality, suggesting a way to analyze and predict the effects of composite prompt operations. We further introduce pointwise mutual information (PMI) vectors to reduce the influence of unconditional distributions; in some cases, PMI-based model maps better reflect training-data-related differences. Overall, the framework supports the analysis of input-dependent model behavior.
