Search papers, labs, and topics across Lattice.
This paper investigates the performance degradation of learned sparse retrieval (LSR) models, particularly SPLADE, when using large pretrained encoders with high MLM-head L2 norms. The authors identify a scale mismatch in the MLM head as the cause of instability and poor performance, leading to distorted matching scores and training collapse. By introducing a simple initialization-time correction that rescales the MLM-head projection, they achieve significant improvements in training stability and retrieval effectiveness, even surpassing traditional BERT-SPLADE baselines in some cases.
A simple rescaling of the MLM-head can turn unstable training runs into competitive sparse retrieval models, challenging the notion that bigger encoders alone drive performance.
Learned sparse retrieval (LSR) models such as SPLADE have traditionally used BERT-style masked language models as backbone encoders. A natural expectation is that replacing BERT with stronger pretrained encoders should improve retrieval effectiveness. However, we find that under standard SPLADE training recipes, backbones with large MLM-head L2 norms can suffer performance degradation and even training collapse under standard SPLADE training recipes. We identify this failure as a scale mismatch in the MLM head: SPLADE directly uses MLM-head outputs to construct sparse lexical representations, and query-document relevance is computed by an unnormalized dot product over these representations. As a result, an inflated MLM-head scale can amplify sparse activations, distort matching scores, and destabilize contrastive training under common training settings. To address this issue, we introduce a simple initialization-time correction that rescales the MLM-head projection by a constant factor before SPLADE training. This zero-cost adjustment improves training stability without modifying the model architecture or training objective. Across both in-domain and out-of-domain retrieval benchmarks, this simple correction substantially improves large-norm backbones such as ModernBERT and Ettin, turning unstable training runs into competitive sparse retrievers. In several settings, the corrected models further match or surpass the classic BERT-SPLADE baseline. These findings suggest that the bottleneck in adapting pretrained encoders to LSR is not encoder capacity alone, but the calibration of the MLM-head scale used to construct sparse lexical representations.
