Search papers, labs, and topics across Lattice.
This paper extends the Hypencoder retrieval approach by introducing the Matryoshka Hypencoder, which utilizes multiple sizes of shallow neural networks (Q-Nets) to balance effectiveness and efficiency. The authors demonstrate that this new architecture achieves comparable effectiveness in-domain while significantly reducing the number of active parameters—by approximately 7× in-domain and 50% out-of-domain—resulting in a 1.6–3.4× increase in scoring throughput. These findings highlight the potential for more efficient retrieval systems in practical applications, making Hypencoders more deployable in real-world scenarios.
Achieving 7× fewer parameters while maintaining effectiveness could revolutionize the efficiency of retrieval systems in AI applications.
The Hypencoder is a recently-proposed retrieval approach that encodes queries as shallow neural networks (''Q-Nets'') that estimate relevance over pre-computed document embeddings. Inspired by Matryoshka Representation Learning, we show that the Hypencoder can be extended to support multiple sizes of Q-Nets, allowing trade-offs between effectiveness and efficiency when deployed. We find that this ''Matryoshka Hypencoder'' achieves comparable in-domain effectiveness with approximately 7× fewer active parameters in-domain and half as many active parameters out-of-domain, which corresponds to a 1.6–3.4× increase in scoring throughput. This work paves the way for practical deployment of Hypencoders. https://github.com/MajdAlkawaas/hypencoder-paper
