Search papers, labs, and topics across Lattice.
This paper introduces Memory-Supported Synergistic Adaptation (MSSA), a training-free test-time adaptation (TTA) framework designed for medical image segmentation that addresses the challenges of distribution shifts without updating model parameters. By leveraging a noise-aware memory construction module and a relevance-driven prototype alignment module, MSSA effectively stabilizes cross-modal predictions and enhances adaptation through reliable text-guided semantic priors. Extensive evaluations across multiple medical segmentation benchmarks reveal that MSSA significantly outperforms traditional fine-tuning-based TTA methods, achieving improvements of up to 12.2% in Dice Similarity Coefficient (DSC) and 11.7% in mean Intersection over Union (mIoU).
Achieving up to 12.2% improvement in segmentation accuracy without any model parameter updates could revolutionize how we handle distribution shifts in medical imaging.
Test-time adaptation (TTA) aims to mitigate distribution shifts by adapting models with unlabeled target data at inference time. While TTA with vision-language models (VLMs) has shown promising results in classification, extending it to medical image segmentation remains challenging. In this setting, the adaptation gains from optimizing on VLM-generated predictions are often outweighed by the degradation to the VLM's strong pretrained features caused by noisy, update-driven learning, resulting in limited and unstable improvements. We therefore propose Memory-Supported Synergistic Adaptation (MSSA), a novel training-free TTA framework for medical image segmentation. Without updating model parameters, MSSA dynamically selects reliable image-text predictions to construct an online memory, uses them as text-guided semantic priors, and couples them with cross-image structural alignment for robust adaptation. Specifically, MSSA consists of (i) a noise-aware memory construction module that filters and stabilizes cross-modal predictions, and (ii) a relevance-driven prototype alignment module that aligns the target sample with structurally consistent memory samples and their reliable predictions to improve adaptation. Extensive experiments on multiple medical segmentation benchmarks demonstrate that MSSA consistently improves VLM-based segmentation models and outperforms existing fine-tuning-based TTA methods by a clear margin, with gains of up to 12.2% DSC and 11.7% mIoU. Project page: https://lingrayy.github.io/MSSA/ .
