Search papers, labs, and topics across Lattice.
The paper introduces Dynamic Dense Retrieval (DDR), a novel approach to address the limitations of fine-tuning dense retrieval (DR) models for new tasks, specifically limited data availability and the need for frequent model updates. DDR employs prefix tuning to create domain-specific modules that are dynamically combined using a routing strategy. Experiments on six zero-shot downstream tasks demonstrate that DDR outperforms traditional DR while using only 2% of the training parameters.
Forget full fine-tuning: this dynamic routing strategy lets you adapt dense retrieval to new domains while using just 2% of the parameters.
The \textit{de facto} paradigm for applying dense retrieval (DR) to new tasks involves fine-tuning a pre-trained model for a specific task. However, this paradigm has two significant limitations: (1) It is difficult adapt the DR to a new domain if the training dataset is limited. (2) Old DR models are simply replaced by newer models that are trained from scratch when the former are no longer up to date. Especially for scenarios where the model needs to be updated frequently, this paradigm is prohibitively expensive. To address these challenges, we propose a novel dense retrieval approach, termed \textit{dynamic dense retrieval} (DDR). DDR uses \textit{prefix tuning} as a \textit{module} specialized for a specific domain. These modules can then be compositional combined with a dynamic routing strategy, enabling highly flexible domain adaptation in the retrieval part. Extensive evaluation on six zero-shot downstream tasks demonstrates that this approach can surpass DR while utilizing only 2\% of the training parameters, paving the way to achieve more flexible dense retrieval in IR. We see it as a promising future direction for applying dense retrieval to various tasks.
Mila
UW