Search papers, labs, and topics across Lattice.
This paper introduces Reduced Matrix Multiplication (RMM), an input-adaptive inference method that optimizes matrix products in Transformer models by selecting informative slices without altering model weights. RMM achieves a predictable accuracy-efficiency trade-off across various model sizes and tasks, demonstrating that reduction tolerance is influenced by model family and retention ratio, often improving with larger models. Mechanistic analyses reveal that attention-side computations are more amenable to reduction than MLP components, leading to significant runtime improvements, particularly for longer sequences.
RMM reveals that optimizing attention-side computations can lead to substantial runtime gains in Transformer models without sacrificing accuracy.
Transformer-based language models achieve strong performance but incur substantial inference cost due to repeated high-dimensional matrix multiplications. We propose Reduced Matrix Multiplication (RMM), a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along their contraction dimensions, without modifying model weights. Under a simple retention-ratio control, RMM provides a smooth and predictable accuracy-efficiency trade-off. Across language models ranging from 1B to 70B parameters, we find that reduction tolerance depends on the model family, task, component, and retention ratio, although it often improves with model scale. Under moderate reduction, RMM remains robust across the evaluated discriminative, autoregressive generation, and long-context settings. We further show that the same principle extends to multimodal vision-language inference. Mechanistic ablations reveal a structural asymmetry within Transformers: attention-side computations are substantially more reducible than MLP components. Finally, wall-clock benchmarks with custom kernels on an NVIDIA A100 show that these computational savings can translate into practical runtime gains, especially at longer sequence lengths. Together, these results position RMM as a scalable direction for input-adaptive inference-time optimization.
