Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
LLM2Vec-Gen is introduced as a self-supervised method for learning generative text embeddings from LLMs by optimizing special tokens appended to the input to represent the LLM's response to a query. Training leverages the LLM's own completions and an unsupervised embedding teacher for distillation, enabling the transfer of LLM capabilities like safety alignment and reasoning to embedding tasks. The approach achieves state-of-the-art self-supervised performance on MTEB, with significant improvements in harmful content retrieval and reasoning capabilities, while also producing interpretable embeddings.
Forget contrastive learning: LLM2Vec-Gen learns text embeddings by representing the *response* an LLM would generate, unlocking safety and reasoning abilities for embedding tasks.
LLM-based text embedders typically encode the semantic content of their input. However, embedding tasks require mapping diverse inputs to similar outputs. Typically, this input-output is addressed by training embedding models with paired data using contrastive learning. In this work, we propose a novel self-supervised approach, LLM2Vec-Gen, which adopts a different paradigm: rather than encoding the input, we learn to represent the model's potential response. Specifically, we add trainable special tokens to the LLM's vocabulary, append them to input, and optimize them to represent the LLM's response in a fixed-length sequence. Training is guided by the LLM's own completion for the query, along with an unsupervised embedding teacher that provides distillation targets. This formulation helps to bridge the input-output gap and transfers LLM capabilities such as safety alignment and reasoning to embedding tasks. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 9.3% over the best unsupervised embedding teacher. We also observe up to 43.2% reduction in harmful content retrieval and 29.3% improvement in reasoning capabilities for embedding tasks. Finally, the learned embeddings are interpretable and can be decoded into text to reveal their semantic content.
Mila