Search papers, labs, and topics across Lattice.
This paper introduces Document Embedding Preservation Tuning (DEPT), a novel approach that trains a single decoder-only LLM to simultaneously generate query expansions and encode document representations, addressing the limitations of traditional methods that decouple these tasks. By maintaining the proximity of tuned document embeddings to their initial states while allowing retrieval gradients to inform query expansion, DEPT effectively navigates the challenges posed by joint query-document movement. Experiments demonstrate that DEPT significantly enhances retrieval quality across multiple datasets compared to existing training-free and staged baselines, showcasing its potential for improved query expansion and retrieval alignment.
DEPT achieves superior retrieval quality by preserving document embeddings while adapting query expansions, revealing a new synergy between these traditionally separate tasks.
Large language models (LLMs) can both expand underspecified queries and encode text as dense representations, suggesting a unified model for query expansion and retrieval. Existing systems usually rely on prompted expansions, independently trained modules, or staged optimization, leaving generated expansions only indirectly aligned with the retrieval loss that judges them. We train a single decoder-only LLM end to end, where the same model generates the expansion and encodes both the expanded query and candidate documents. This unified setting creates a moving-target problem: retrieval supervision should improve query-side expansion, but the same update also shifts the document embeddings that serve as retrieval targets. We introduce Document Embedding Preservation Tuning (DEPT), which keeps tuned document embeddings close to cached initial embeddings while allowing retrieval gradients to pass through straight-through decoding into the generator. DEPT converts joint query--document movement into query-side adaptation against approximately stable, whitened document embeddings that support index reuse and online hard-negative mining. Experiments with Qwen3-4B-Instruct-2507 and LLaMA-3.2-3B-Instruct on five datasets in BEIR benchmark show that DEPT improves average retrieval quality over training-free, independently trained, and staged unified baselines, while ablations isolate the effects of preservation, whitening, end-to-end expansion training, and online negatives. Code is available at https://github.com/ILSparkle/DEPT.
