Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces $σ$N-Ens, a normalization-based implicit ensemble that enhances member diversity during training by treating each member as a task within a multi-task architecture and utilizing sigmoid-bounded scalers for modulation. The method achieves significant improvements in uncertainty estimation while maintaining lower parameter costs compared to traditional deep ensembles, which typically require linear scaling with the number of members. Evaluations on datasets such as CIFAR-10/100 and ImageNet demonstrate that $σ$N-Ens not only matches or surpasses the performance of deep ensembles but also maintains calibration under distribution shifts.
$σ$N-Ens achieves deep ensemble-level performance with significantly reduced parameter costs and improved calibration under distribution shifts.
Deep ensembles provide the most reliable uncertainty estimates in deep learning, but their cost grows linearly with the number of members. Implicit ensembles lower this cost by sharing a single backbone across members. Member diversity is a primary determinant of ensemble quality, yet no implicit ensemble can shape it during training; existing methods fix it at initialisation or build it into the architecture. We introduce $σ$N-Ens, a normalisation-based implicit ensemble that treats each member as a task in a multi-task architecture and modulates the shared backbone through sigmoid-bounded scalers. We also introduce a softmax-temperature regulariser, which shapes the equilibrium level of sharing between members and traces the accuracy-calibration frontier. Because only normalisation layers are replicated, the mechanism can wrap convolutional and transformer backbones alike, also allowing pretrained models to be adapted through a short fine-tune. We frame the epistemic uncertainty such an ensemble expresses as modulation uncertainty, and explain why its calibration holds under input corruption, and why its out-of-distribution detection is weaker. Our method is evaluated across ResNets and transformers on CIFAR-10/100, ImageNet and SST-2. $σ$N-Ens matches or outperforms deep ensembles at a fraction of their parameter cost, scales with ensemble size where partitioning methods collapse, and maintains calibration under distribution shift.