Search papers, labs, and topics across Lattice.
This paper introduces Distance-Adaptive Representation (DAR), a novel approach for optimizing key and value dimensionality in decoder-only Transformers by assigning richer representations to local tokens while using lower-dimensional representations for distant tokens. The method was tested across various pretraining scales and demonstrated that it closely matches the performance of full-dimensional baselines, unlike uniform dimensionality reduction, which degraded performance. These findings challenge the traditional assumption of uniform dimensionality in attention mechanisms and suggest a more efficient way to manage representational capacity in language models.
Local tokens deserve richer representations, while distant tokens can get by with less鈥攖his insight could transform attention architecture design.
Decoder-only Transformers compute attention over the KV cache of preceding tokens. Keys (and Values) are typically represented with the same dimensionality, regardless of its distance from the prediction target. In natural language, however, the next word is most strongly influenced by the immediately preceding tokens. We hypothesize that local and distant tokens impose asymmetric demands on representational capacity: local tokens are more critical for predicting immediate outputs and thus require richer representations, whereas distant tokens primarily serve as long-range memory, for which lower-dimensional representations may suffice. We formalize this idea as Distance-Adaptive Representation (DAR), implemented in a controlled setting that preserves full-dimensional representations within a local context window while assigning reduced-dimensional representations (e.g. 1/4 of the original dimensionality) to tokens beyond that window. Across multiple pretraining scales (70M to 410M parameters), as well as continued supervised fine-tuning on a 1B-scale model, this approach closely matches the performance of full-dimensional baselines. In contrast, uniformly reducing dimensionality across all token positions leads to worse performance. These results challenge the common assumption that key and value dimensionality should be uniform across token positions. Our findings suggest a new direction for designing attention architectures that adaptively allocate representational capacity across sequences, enabling further reductions in KV cache during inference.
