Search papers, labs, and topics across Lattice.
This paper introduces PartRep, a selective augmentation method for decoder-only LLMs that addresses the issue of asymmetric information flow in causal attention by appending only the most informative tokens from the prompt rather than repeating the entire prompt. By employing token-wise negative log-likelihood as a selection signal, PartRep effectively redistributes contextual grounding, enhancing reasoning performance without incurring the high computational costs associated with full prompt repetition. The method demonstrates significant efficiency, achieving 59.4% of the KV cache usage and 79.0% of the prefill FLOPs of full repetition while retaining most performance gains across multiple benchmarks and model families.
Selectively repeating only the most informative tokens can dramatically enhance reasoning in LLMs while slashing computational costs.
While decoder-only LLMs excel at a vast array of natural language tasks, it suffers from an asymmetric information flow induced by causal attention: later tokens are richer in contextual grounding than earlier ones. A simple and effective remedy is prompt repetition -- just appending a second copy of prompt before generation can redistribute grounding across positions and improve reasoning performance. However, full repetition of the original prompt doubles the KV cache footprint and quadruples attention cost during prefill, making it impractical for long-context settings. We propose PartRep, a selective augmentation method that appends only the most informative tokens -- rather than the entire prompt. We use token-wise negative log-likelihood (NLL) as a selection signal, motivated by the hypothesis that less predictable tokens are less recoverable from surrounding context and therefore benefit more from late-position repetition. To avoid the heavy cost of a full forward pass for scoring, we train a lightweight gate that predicts high-NLL tokens from early-layer hidden states, enabling token selection during mid-prefill via early exit. Across eight benchmarks (including MMLU, GSM8K, and RULER) and three model families (Qwen2.5, Llama3.2, Gemma4), PartRep retains most of the gains of full repetition while using only 59.4\% of its KV cache and 79.0\% of its prefill FLOPs.
