Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces PUMA, a novel approach for post-hoc sparsification of universal multimodal embeddings aimed at enhancing retrieval efficiency without retraining the backbone model. By employing a sparse autoencoder that fine-tunes the embeddings for retrieval while preserving dense dot-product geometry, PUMA achieves significant reductions in vector storage and inference time. The method demonstrates competitive performance against dense retrieval on multiple benchmarks, achieving up to 25x speed improvements and 8-16x reductions in storage requirements.
PUMA slashes memory usage by up to 16x and boosts retrieval speed by 25x, all while maintaining competitive performance against dense retrieval methods.
Universal multimodal embedders enable retrieval across text, image, and combined queries, but their dense representations incur high memory and inference costs. Post-hoc sparsification could reduce these costs but remains underexplored for multimodal retrieval. We introduce PUMA, a sparse autoencoder recipe that maps universal multimodal embeddings to compact sparse codes without retraining the backbone: a pretraining stage preserves dense dot-product geometry, after which the sparse encoder is fine-tuned for retrieval. We evaluate on five benchmarks covering text-to-image and composed image retrieval. On Qwen3-VL-Embedding-2B, PUMA is statistically indistinguishable from or improves over dense retrieval on four of five datasets. We further identify two failure modes of post-hoc sparsification: insufficient pre-TopK support and retrieval-misaligned active support. PUMA reduces vector storage by 8-16x (FP32) and is up to 25x faster than exact dense scoring on larger candidate pools, enabling efficient multimodal retrieval.