Search papers, labs, and topics across Lattice.
This paper introduces Multi-Prefix Embedding (MPE), a novel approach to long-context retrieval that balances the trade-off between detailed representation and storage efficiency by encoding document chunks in a single causal forward pass. MPE effectively retains cross-chunk context and facilitates chunk-level MaxSim matching using only document-level relevance labels. Experimental results demonstrate that MPE not only competes with but often outperforms existing embedding methods, while also offering a mechanism for evidence chunk attribution.
MPE achieves superior long-context retrieval by efficiently encoding document chunks while maintaining critical contextual relationships, outperforming traditional methods.
Long-context retrieval exposes a tension: single-vector embeddings lose fine-grained detail, while token-level multi-vector methods incur prohibitive storage. We propose Multi-Prefix Embedding (MPE), which partitions a document into chunks separated by EOS tokens, encodes the full sequence in a single causal forward pass, and extracts one embedding at each prefix boundary. MPE retains cross-chunk context, enables chunk-level MaxSim matching, and trains with only document-level relevance labels. Experiments on MLDR-en, BrowseComp-Plus, and LongEmbed show that MPE is competitive with or outperforms single-vector, independent-chunk, and multi-vector baselines, while providing a natural source attribution mechanism for locating evidence chunks.
CMU ML