Search papers, labs, and topics across Lattice.
Investigating depth-dependent prefix redundancy in decoder-based embedding models like Qwen3-Embedding, this work identifies that token representations become increasingly compressible at deeper layers without harming the final readout representation. The authors leverage this dynamic with FastE, a training-free inference framework that triggers mid-network token pruning based on online readout-prefix alignment and attention attribution. On long-context benchmarks like NarrativeQA, FastE slashes decoder FLOPs by 40.11% while preserving 99.53% of baseline nDCG@10 across text and cross-modal retrieval setups.
LLM embedding models waste over 40% of their inference compute processing prefix tokens that become completely redundant at deeper layers, enabling aggressive, training-free token pruning with virtually zero retrieval degradation.
In this study, we identify depth-dependent prefix redundancy in final-readout LLM embedding models, notably across representative backbones including Qwen3-Embedding and Qwen3-VL-Embedding. We find that removing prefix states is substantially more damaging in shallow layers than at greater depth, showing that prefix states become increasingly compressible as the prefix and readout states propagate through the network. To this end, we introduce FastE, a training-free, plug-and-play method. FastE uses a shared fixed threshold on batch-mean readout-prefix alignment as a lightweight online heuristic for selecting when compression occurs, and ranks prefix states by the attention scores they receive from the readout position to determine which states are retained in subsequent layers. Our evaluations demonstrate FastE's ability to substantially reduce computational costs: on NarrativeQA with Qwen3-Embedding-0.6B, it reduces decoder-backbone FLOPs by 40.11% while retaining 99.53% of Full Forward nDCG@10. Across five text embedding benchmarks, two backbone scales, and three cross-modal retrieval tasks, the quality-efficiency trade-off is directly customizable through the maximum removal ratio without retraining. We believe FastE offers practical value for scalable embedding generation in retrieval, indexing, clustering, and multimodal representation systems.
