Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Tensor Mixture (MixT), a general tensor-structured compression scheme that replaces dense linear layers in LLMs with mixtures of tensor operators. MixT operates directly on generic linear projections, making it applicable across various Transformer-based LLMs. Experiments on Qwen3-8B and LLaMA2-7B show that MixT can significantly reduce model parameters (47.5%), inference FLOPs (37.1%), training FLOPs (52.1%), and peak inference memory (60.4%) in LLaMA2-7B while largely preserving MMLU accuracy up to a model-specific compressibility boundary.
LLMs can be compressed by nearly 50% with minimal accuracy loss using a new tensor decomposition method, but pushing further reveals abrupt, model-specific compressibility limits tied to shifts in internal representations.
Large language models (LLMs) are dominated by dense linear transformations, whose storage, memory and computational overheads hinder efficient adaptation and deployment while masking the functional impacts of structural simplification. Here we present Tensor Mixture (MixT), a general tensor-structured compression scheme that replaces targeted dense linear layers with natively executable mixtures of tensor operators. Operating directly on generic linear projections instead of model-specific components, MixT is potentially applicable across Transformer-based LLMs and other dense neural mappings. We evaluate MixT on Qwen3-8B and LLaMA2-7B under a unified recovery protocol, identifying a broad compressible regime in which MMLU accuracy is largely preserved before an abrupt transition at model-specific boundaries. This transition coincides with coordinated shifts in output entropy, prediction entropy and inter-layer geometry. At the LLaMA2-7B transition boundary, MixT reduces full-model parameters by 47.5\%, inference FLOPs by 37.1\%, training FLOPs by 52.1\% and peak inference memory by 60.4\%, demonstrating its practical potential for lower-cost LLM compression.