Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces DRIFT, a novel framework that adapts pretrained vision-language models (VLMs) for continuous output tasks by combining a base predictor with a generative refinement module based on flow matching. The approach addresses the limitations of traditional autoregressive decoding in VLMs, which struggle with tasks requiring precise continuous outputs, such as event localization and robotic control. Evaluation across various perception and planning tasks demonstrates that DRIFT significantly outperforms existing regression and generative methods, showcasing its effectiveness in enhancing VLM capabilities for continuous decoding.
DRIFT transforms the generative modeling landscape by simplifying the optimization of continuous outputs, leading to superior performance in both perception and robotic control tasks.
Many modern vision-language models (VLMs) build on autoregressive decoding of discrete tokens. While text-based output interfaces enable scalable pretraining and strong zero-shot generalization across diverse tasks, they are poorly suited for problems that require precise continuous outputs, such as localizing temporal boundaries of events or generating robotic control actions. To address this challenge, we propose DRIFT, a general framework for adapting pretrained VLMs to continuous decoding tasks. DRIFT combines a base predictor, which provides a coarse estimate of the target output, with a generative refinement module based on flow matching that iteratively improves the prediction. This residual formulation transforms the generative modeling problem from learning a global output distribution to modeling a localized residual distribution around a strong prior, substantially simplifying optimization. We evaluate DRIFT on both perception and planning tasks, including visual grounding and robotic control. Across multiple tasks and architectures spanning MLLMs, VLAs, and WAMs, DRIFT consistently outperforms a strong set of regression- and generative-based solutions.