Search papers, labs, and topics across Lattice.
This paper introduces MAVISEG, a training-free refinement layer for zero-shot open-vocabulary semantic segmentation in diffusion transformers, which leverages structured signals from the generative process that traditional attribution methods overlook. By utilizing a pixel-by-concept score field and pixel feature space, MAVISEG demonstrates superior performance across six benchmarks, achieving the highest mean Intersection over Union (mIoU) scores among training-free methods. Notably, the method excels particularly in scenarios where initial segmentation is weakest, revealing that diffusion transformers retain more concept-level information than previously recognized.
MAVISEG reveals that diffusion transformers can retain and utilize more structured visual information than traditional methods, leading to significant improvements in segmentation accuracy.
Text-to-image diffusion transformers learn about objects and scenes by learning to generate them, making them strong candidates for training-free zero-shot open-vocabulary semantic segmentation. State-of-the-art attribution methods score each pixel independently, comparing its features against a fixed text-derived class representation, whether as an output-space similarity or as a cross-attention weight. This discards structured signals the model itself exposes: the temporal structure of the generative trajectory, the visual appearance statistics of each concept, and the image's own pairwise feature geometry. We present MAVISEG, a training-free refinement layer that recovers these signals. Because its operators consume only a pixel-by-concept score field and a pixel feature space, MAVISEG is capture-agnostic rather than tied to one attribution method. Across six benchmarks it achieves the strongest overall results among training-free methods, including the best mIoU on every benchmark. Interestingly, gains are largest where the initial capture is weakest, and individual operators contribute depending on the noise in the field they refine. Our results indicate that diffusion transformers carry more concept-level information than current attribution methods recover, and that much of it is lost on the way to the mask rather than absent from the model.
