Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces "partial fusion," a novel method for interpolating between neural network ensembles and weight aggregation to balance performance and computational cost. Partial fusion selectively aggregates weights of similar neurons across networks, identified using a joint similarity measure and partial optimal transport. The authors also frame weight aggregation and partial fusion as generalized pruning, where neurons can be deleted or linearly combined, and demonstrate its effectiveness even within a single network.
Achieve ensemble-level performance at a fraction of the computational cost by selectively fusing similar neurons across networks.
Ensembles of neural networks typically outperform individual networks but incur large computational costs, whereas weight aggregation produces less costly, yet also less accurate, aggregate models. We introduce partial fusion of networks, which interpolates between ensembles and weight aggregation and thus allows for a flexible tradeoff between computational cost and performance. A direct way to achieve this is to extend existing weight aggregation methods based on neuron-level similarity between different networks, where partial fusion then only aggregates weights of neurons which are most similar. We showcase one particular method to jointly identify which neurons are most similar and match them via partial optimal transport. Further, we consider the more general perspective of weight aggregation and partial fusion as generalized pruning of ensemble models, where neurons cannot just be deleted, but also linearly combined. Finally, we show that generalized pruning applied to a single network yields similar benefits as partial fusion by allowing for a tradeoff between isolating, deleting, and linearly combining neurons based on similarity. Our code is available at https://github.com/Fabian-Mor/partial_fusion_nn.