Search papers, labs, and topics across Lattice.
SwInception enhances the Swin Transformer for medical image segmentation by incorporating Inception blocks within the feed-forward network layers to improve multi-scale feature extraction. This modification increases the inductive bias of the transformer, mitigating overfitting on small datasets. Experiments across eleven medical datasets, including the Medical Segmentation Decathlon, demonstrate performance improvements over state-of-the-art backbones.
Inception-style convolutions can significantly boost the performance of Swin Transformers on medical image segmentation, challenging the assumption that attention alone is sufficient for optimal performance.
Sparse vision transformers have gained popularity as efficient encoders for medical volumetric segmentation, with Swin emerging as a prominent choice. Swin uses local attention to reduce complexity and yields excellent performance for many tasks but still tends to overfit on small datasets. To mitigate this weakness, we propose a novel architecture that further enhances Swin's inductive bias by introducing Inception blocks in the feed-forward layers. The introduction of these multi-branch convolutions enables more direct reasoning over local, multi-scale features within the transformer block. We have also modified the decoder layers in order to capture finer details using fewer parameters. We demonstrate a performance improvement on eleven different medical datasets through extensive experimentation. We specifically showcase advancements over the previous state-of-the-art backbones on benchmark challenges like the Medical Segmentation Decathlon and Beyond the Cranial Vault. By showing that the existing inductive bias in Swin can be further improved, our work presents a promising avenue for enhancing the capabilities of sparse vision transformers for both medical and natural image segmentation tasks. Code and pre-trained weights can be accessed at https://github.com/Eiphodos/SwInception.
