Search papers, labs, and topics across Lattice.
The paper introduces Token-UNet, a 3D segmentation model that integrates Transformers into UNets using TokenLearner and TokenFuser modules to improve efficiency and interpretability. By maintaining a convolutional encoder and applying TokenLearner to 3D feature maps, the model reduces computational demands compared to SwinUNETR. Experiments demonstrate that Token-UNet achieves a better average Dice score (87.21%) with significantly reduced memory footprint (33%), computation time (10%), and parameter count (35%) compared to SwinUNETR.
Achieve state-of-the-art 3D medical image segmentation with 10x faster inference and 3x smaller memory footprint by strategically integrating Transformers into UNets.
We present Token-UNet, adopting the TokenLearner and TokenFuser modules to encase Transformers into UNets. While Transformers have enabled global interactions among input elements in medical imaging, current computational challenges hinder their deployment on common hardware. Models like (Swin)UNETR adapt the UNet architecture by incorporating (Swin)Transformer encoders, which process tokens that each represent small subvolumes ($8^3$ voxels) of the input. The Transformer attention mechanism scales quadratically with the number of tokens, which is tied to the cubic scaling of 3D input resolution. This work reconsiders the role of convolution and attention, introducing Token-UNets, a family of 3D segmentation models that can operate in constrained computational environments and time frames. To mitigate computational demands, our approach maintains the convolutional encoder of UNet-like models, and applies TokenLearner to 3D feature maps. This module pools a preset number of tokens from local and global structures. Our results show this tokenization effectively encodes task-relevant information, yielding naturally interpretable attention maps. The memory footprint, computation times at inference, and parameter counts of our heaviest model are reduced to 33\%, 10\%, and 35\% of the SwinUNETR values, with better average performance (86.75\% $\pm 0.19\%$ Dice score for SwinUNETR vs our 87.21\% $\pm 0.35\%$). This work opens the way to more efficient trainings in contexts with limited computational resources, such as 3D medical imaging. Easing model optimization, fine-tuning, and transfer-learning in limited hardware settings can accelerate and diversify the development of approaches, for the benefit of the research community.
