Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper identifies a novel failure mode in iterative segmentation pipelines using models like SAM, termed "decoder coupling drift," where the mask decoder's attention progressively misaligns with the target object due to feedback loops. They instrument SAM's decoder to measure prompt-image coupling, attention stability, and temporal consistency, revealing degradation compared to oracle-anchored feedback. To mitigate this, they propose DeCoDrift, a training-free inference-time stabilization framework that constrains prompt updates, improving attention stability, temporal coherence, and segmentation quality.
SAM's iterative segmentation can go haywire due to "decoder coupling drift," but a simple inference-time fix called DeCoDrift can bring it back on track without retraining.
Foundation segmentation models such as Segment Anything Model (SAM) are now routinely used in iterative pipelines, where each predicted mask is fed back as the next prompt. This practice turns segmentation into a closed-loop dynamical process, yet the decoder-level behavior of these systems remains largely unexamined. We show that this feedback loop can induce a previously overlooked failure mode, decoder coupling drift, in which the mask decoder's cross-attention progressively loses alignment with the target object, causing errors to accumulate across iterations. We study this phenomenon by instrumenting SAM's mask decoder and deriving ground-truth-free measures of prompt-image coupling, attention stability, and temporal consistency. On volumetric electron microscopy data, these decoder-internal signals reveal that standard iterative prompting systematically degrades attention alignment and temporal coherence relative to oracle-anchored feedback. We then formalize iterative prompting as a discrete-time dynamical system and show how proximal anchoring reduces error amplification in the feedback loop. Building on this analysis, we introduce DeCoDrift, a training-free inference-time stabilization framework that constrains prompt updates and preserves decoder coupling across iterations. Across extensive experiments, DeCoDrift consistently improves attention stability, temporal coherence, and segmentation quality over standard iterative prompting, without retraining or ground-truth supervision. More broadly, our results show that decoder-internal dynamics are not merely diagnostic: they provide actionable signals for stabilizing foundation segmentation models in closed-loop use.