Search papers, labs, and topics across Lattice.
Argus-Unified introduces a compact multimodal model that unifies visual understanding and generation while significantly reducing computational and data demands. By leveraging pretrained vision-language models and employing hybrid visual tokens, the model achieves state-of-the-art performance on multimodal understanding benchmarks and competitive generation quality. Remarkably, it accomplishes this with only 15.6M data points and a training cost of approximately $2,000, making it an economically viable alternative to existing models.
Achieving state-of-the-art multimodal understanding at a fraction of the cost and data requirements challenges the notion that bigger models are always better.
Unifying visual understanding and generation in one model holds immense promise, but remains challenging and expensive due to heavy compute and data demands and conflicts between the visual features needed for these two capabilities. To address these challenges, we present Argus-Unified, a compact, effective and unified multimodal model built with low demand on computation and data. Instead of aligning modalities from scratch, Argus-Unified effectively leverages pretrained vision-language models (VLMs) that provide strong multimodal priors. Specifically, we introduce hybrid visual tokens that preserve continuous tokens for understanding while learning discrete tokens for generation from a frozen unified vision encoder. Our training pipeline includes two stages: the first stage learns a quantizer and image decoder on top of the frozen vision encoder, the second stage trains the LLM initialized from a pretrained VLM for the unified multimodal modeling. Using by far the least amount of data (15.6M) and the lowest cost (~$2,000), we demonstrate that unified multimodal models can be trained economically while achieving strong performance in both understanding and generation. Notably, our model attains state-of-the-art multimodal understanding on GQA, POPE, and VQAv2, and competitive generation quality compared to models with dedicated vision encoders (e.g., Janus, Janus-Pro), all at ~10x lower cost and with ~5x less data. We envision Argus-Unified as a useful baseline that lowers the development barrier for unified models.
