Search papers, labs, and topics across Lattice.
The paper introduces STORM, a self-supervised framework for lexical query expansion that optimizes query rewriting through a reward-guided beam search mechanism. By scoring candidate expansions against a BM25 index and pruning low-reward options, STORM provides a token-level signal that enhances retrieval effectiveness without the need for extensive model retraining. Experimental results demonstrate that STORM achieves competitive performance with LLM rewriters while maintaining the efficiency of traditional BM25, even outperforming dedicated multilingual dense retrievers across multiple languages.
STORM transforms lexical query expansion by turning retrieval rewards into actionable token-level signals, enabling efficient and effective query rewriting that rivals larger models.
Modern retrieval increasingly relies on dense and learned-sparse neural models that are effective but require encoding the entire corpus into a specialized index, rebuilt whenever the model changes. Lexical retrievers like BM25 stay efficient and transparent on a standard inverted index that need not change as models evolve, but suffer from vocabulary mismatch. LLM query rewriting can help, yet prompted rewriters emit well-formed but retrieval-ineffective or harmful-terms, and training against a retrieval reward gives only delayed, sequence-level supervision that obscures which terms helped. We introduce STORM (Stepwise Token Optimization with Reward-guided beaM search), a self-supervised framework for lexical query expansion. STORM trains the rewriter through generation guided by retrieval metrics: at each step, candidate expansions are scored against the BM25 index and low-reward continuations pruned, turning the retrieval reward into a token-level signal that concentrates exploration on retrieval-effective vocabulary. Across TREC DL and BEIR, STORM lets 0.6B-8B backbones match or surpass competitive LLM rewriters while retrieving as fast as plain BM25; at 8B it rivals far larger proprietary rewriters. It further transfers zero-shot to 18 languages (MIRACL), beating dedicated multilingual dense retrievers on average, making STORM a competitive, infrastructure-light alternative to dense neural retrieval.
Mila