Search papers, labs, and topics across Lattice.
Wallaroo, a new autoregressive model, unifies multi-modal understanding, image generation, and editing through next-token prediction. The model incorporates decoupled visual encoding pathways and is trained using a four-stage strategy to enhance its capabilities. Experiments across various benchmarks demonstrate Wallaroo's competitive performance, highlighting the potential of autoregressive models for multi-modal tasks.
A surprisingly simple autoregressive model rivals specialized architectures in multi-modal understanding, generation, and editing.
In this work, we introduce Wallaroo, a simple autoregressive baseline that leverages next-token prediction to unify multi-modal understanding, image generation, and editing at the same time. Moreover, Wallaroo supports multi-resolution image input and output, as well as bilingual support for both Chinese and English. We decouple the visual encoding into separate pathways and apply a four-stage training strategy to reshape the model's capabilities. Experiments are conducted on various benchmarks where Wallaroo produces competitive performance or exceeds other unified models, suggesting the great potential of autoregressive models in unifying multi-modality understanding and generation. Our code is available at https://github.com/JiePKU/Wallaroo.
