Search papers, labs, and topics across Lattice.
This paper introduces PUMA, a novel approach for post-hoc sparsification of universal multimodal embeddings, aimed at reducing the high memory and inference costs associated with dense representations. By employing a sparse autoencoder that fine-tunes embeddings without retraining the backbone, PUMA achieves significant efficiency gains while maintaining competitive retrieval performance across multiple benchmarks. The results show that PUMA can reduce vector storage by 8-16x and improve retrieval speed by up to 25x compared to exact dense scoring, highlighting its practical applicability in multimodal retrieval tasks.
PUMA slashes memory usage by up to 16x while maintaining or enhancing retrieval performance in multimodal tasks.
Universal multimodal embedders enable retrieval across text, image, and combined queries, but their dense representations incur high memory and inference costs. Post-hoc sparsification could reduce these costs but remains underexplored for multimodal retrieval. We introduce PUMA, a sparse autoencoder recipe that maps universal multimodal embeddings to compact sparse codes without retraining the backbone: a pretraining stage preserves dense dot-product geometry, after which the sparse encoder is fine-tuned for retrieval. We evaluate on five benchmarks covering text-to-image and composed image retrieval. On Qwen3-VL-Embedding-2B, PUMA is statistically indistinguishable from or improves over dense retrieval on four of five datasets. We further identify two failure modes of post-hoc sparsification: insufficient pre-TopK support and retrieval-misaligned active support. PUMA reduces vector storage by 8-16x (FP32) and is up to 25x faster than exact dense scoring on larger candidate pools, enabling efficient multimodal retrieval.
