Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores the integration of sparse Mixture-of-Experts (MoE) layers into CNNs for semantic segmentation, focusing on a patch-wise routing strategy. They conduct a design analysis on Cityscapes and BDD100K datasets to assess the impact of architectural choices on routing dynamics and expert specialization. The results show consistent, architecture-dependent improvements (up to +3.9 mIoU) with minimal computational overhead, highlighting the design sensitivity of MoE layers in CNNs.
Patch-wise Mixture-of-Experts can boost CNN semantic segmentation performance by nearly 4 mIoU, but only if you carefully tune the architecture.
Sparse mixture-of-experts (MoE) layers have been shown to substantially increase model capacity without a proportional increase in computational cost and are widely used in transformer architectures, where they typically replace feed-forward network blocks. In contrast, integrating sparse MoE layers into convolutional neural networks (CNNs) remains inconsistent, with most prior work focusing on fine-grained MoEs operating at the filter or channel levels. In this work, we investigate a coarser, patch-wise formulation of sparse MoE layers for semantic segmentation, where local regions are routed to a small subset of convolutional experts. Through experiments on the Cityscapes and BDD100K datasets using encoder-decoder and backbone-based CNNs, we conduct a design analysis to assess how architectural choices affect routing dynamics and expert specialization. Our results demonstrate consistent, architecture-dependent improvements (up to +3.9 mIoU) with little computational overhead, while revealing strong design sensitivity. Our work provides empirical insights into the design and internal dynamics of sparse MoE layers in CNN-based dense prediction. Our code is available at https://github.com/KASTEL-MobilityLab/moe-layers/.