Search papers, labs, and topics across Lattice.
The paper introduces DenseMLLM, a method that enables standard Multimodal Large Language Models (MLLMs) to perform dense prediction tasks like semantic segmentation and depth estimation without task-specific decoders. This is achieved through a novel vision token supervision strategy for multiple labels and tasks, maintaining a generalist MLLM architecture. The resulting DenseMLLM achieves competitive performance on dense prediction and vision-language benchmarks, demonstrating the intrinsic dense prediction capabilities of standard MLLMs.
Standard multimodal LLMs can perform surprisingly well on dense prediction tasks like segmentation and depth estimation, without needing any task-specific decoder modules.
Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in high-level visual understanding. However, extending these models to fine-grained dense prediction tasks, such as semantic segmentation and depth estimation, typically necessitates the incorporation of complex, task-specific decoders and other customizations. This architectural fragmentation increases model complexity and deviates from the generalist design of MLLMs, ultimately limiting their practicality. In this work, we challenge this paradigm by accommodating standard MLLMs to perform dense predictions without requiring additional task-specific decoders. The proposed model is called DenseMLLM, grounded in the standard architecture with a novel vision token supervision strategy for multiple labels and tasks. Despite its minimalist design, our model achieves highly competitive performance across a wide range of dense prediction and vision-language benchmarks, demonstrating that a standard, general-purpose MLLM can effectively support dense perception without architectural specialization.
