Search papers, labs, and topics across Lattice.
This paper investigates the training dynamics of late-interaction retrieval models, focusing on the impact of hard maximum similarity (MaxSim) pooling. Through synthetic experiments and real-world benchmarks, the authors show that MaxSim leads to high gradient concentration and increased sensitivity to document length compared to smoother pooling methods. The study identifies a sparsity-robustness tradeoff inherent in MaxSim, suggesting that alternative pooling strategies may improve the performance and stability of late-interaction retrieval.
MaxSim pooling, a common technique in late-interaction retrieval, suffers from a surprising sparsity-robustness tradeoff, making it brittle to document length variations.
Late-interaction retrieval models rely on hard maximum similarity (MaxSim) to aggregate token-level similarities. Although effective, this winner-take-all pooling rule may structurally bias training dynamics. We provide a mechanistic study of gradient routing and robustness in MaxSim-based retrieval. In a controlled synthetic environment with in-batch contrastive training, we demonstrate that MaxSim induces significantly higher patch-level gradient concentration than smoother alternatives such as Top-k pooling and softmax aggregation. While sparse routing can improve early discrimination, it also increases sensitivity to document length: as the number of document patches grows, MaxSim degrades more sharply than mild smoothing variants. We corroborate these findings on a real-world multi-vector retrieval benchmark, where controlled document-length sweeps reveal similar brittleness under hard max pooling. Together, our results isolate pooling-induced gradient concentration as a structural property of late-interaction retrieval and highlight a sparsity-robustness tradeoff. These findings motivate principled alternatives to hard max pooling in multi-vector retrieval systems.
