Search papers, labs, and topics across Lattice.
Microsoft Researchaka.msBaiduBITFriedrich-Alexander-Universit盲tGeneralAIPKUJun 24, 2026arXiv:2606.25674This paper introduces BITEMBED, an extreme low-bit framework that optimizes LLM-based text embeddings by employing ternary weights and quantized activations to enhance encoding efficiency and reduce storage overhead. Through continual contrastive pre-training and supervised fine-tuning, BITEMBED not only converts pretrained LLMs into efficient embedding encoders but also allows for the generation of output embeddings at multiple storage precisions. Experimental results demonstrate that BITEMBED achieves performance comparable to full-precision models while significantly lowering storage costs, making it a viable solution for large-scale applications.
Achieving comparable performance to full-precision models, BITEMBED slashes storage costs and enhances embedding efficiency with extreme low-bit quantization.
LLM-based text embedders have substantially improved retrieval and semantic representation quality, but their deployment remains costly: large backbone models slow down embedding inference, while high-dimensional full-precision embeddings impose substantial storage and bandwidth overhead on large-scale indexes. In this paper, we present BITEMBED, an extreme low-bit framework for LLM-based text embedding that jointly targets encoding efficiency and vector storage. BITEMBED converts pretrained LLM backbones into BitNet-style embedding encoders with ternary weights, quantized activations, and lightweight normalization refinement. The converted model is adapted to representation learning through continual contrastive pre-training, followed by supervised contrastive fine-tuning with both similarity-distribution distillation and attention-relation distillation from a full-precision teacher. Beyond quantizing the backbone, BITEMBED further trains output embeddings to support multiple storage precisions meeting different storage needs in various scenarios. Experiments on MMTEB (eng, v2) with Qwen3-0.6B and Gemma3-270M show that BITEMBED is largely comparable to full precision teacher embedders. Moreover, BITEMBED flexibly obtains text embeddings of various precisions, achieving a trade-off between performance and storage cost.
