Search papers, labs, and topics across Lattice.
This paper investigates the localization capabilities of Vision-Language Models (VLMs) when used as condition encoders in diffusion-based image editing, revealing a significant performance gap in complex scenes. The authors introduce the Analysis-by-Proxy framework, which trains a lightweight proxy model on the VLM's intermediate representations to uncover how localization information is encoded. Their findings demonstrate that crucial localization signals are often hidden in intermediate representations and do not propagate effectively through existing conditioning architectures, highlighting the need for improved design in editing pipelines.
Localization signals in VLMs remain hidden in intermediate representations, leading to significant failures in current image editing pipelines.
Vision-Language Models (VLMs) are increasingly utilized as the conditioning backbone for diffusion-based image editing due to their remarkable multimodal reasoning capabilities. While standalone VLMs demonstrate strong localization capabilities, editing pipelines frequently struggle to maintain this accuracy, particularly in complex, multi-entity scenes. In this work, we investigate this performance gap, hypothesizing that it stems from treating the VLM as a condition encoder. In this role, the model is restricted to a single forward pass, preventing the autoregressive generation process for which it was optimized, thereby failing to fully expose its capabilities. To investigate whether this spatial understanding persists when the VLM is used as a condition encoder, we introduce Analysis-by-Proxy. In this framework, we train a lightweight, interpretable proxy model on the VLM's intermediate representations using an auxiliary localization task. By analyzing the VLM through this proxy, we uncover the specific VLM representations that encode localization information. Our findings expose a fundamental mismatch between how spatial knowledge is represented within a VLM condition encoder and how it is extracted by current editing pipelines. We reveal that under single-pass constraints, the localization signal does not reliably propagate to the predefined layer configurations commonly used for conditioning. Instead, this crucial signal remains hidden within intermediate representations, at locations that vary depending on the input prompt. Using our introduced Analysis-by-Proxy framework, we reveal the fundamental failures of existing condition extraction strategies in editing pipelines, opening the door to more principled design of conditioning architectures.
