Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the emergence of modular structures in deep neural networks through competitive dynamics among neuron groups during training. By implementing a method that sparsely routes inputs and encourages specialization, the authors achieve near-baseline accuracy while enabling modules to correlate with specific input classes. Evaluations on ImageNet-100 and CIFAR-100 reveal that these specialized modules emerge autonomously, capturing meaningful semantic categories without requiring module-level supervision.
Competitive dynamics among neurons can autonomously induce specialized modules that meaningfully capture semantic categories in neural networks.
Modularity in deep neural networks has been proposed as a means of improving both interpretability and training by promoting disentangled representations and reducing redundancy. In this work, we study the emergence of modular structure through competition dynamics between groups of neurons during training. We introduce a method that (i) maintains near-baseline accuracy, (ii) induces usage-based modularity by sparsely routing inputs to neuron groups, and (iii) encourages specialization of these modules, such that their activations are correlated with input classes. We evaluate the proposed approach on ImageNet-100 and CIFAR-100 and show that with it, specialized modules emerge without module-level supervision. These modules capture a meaningful high-level structure in the data, with individual modules responding to semantic categories (e.g., dogs or vehicles). We also study the emergence of a hierarchical partition of sub-tasks depending on the number of modules. Our results suggest that competitive dynamics can serve as a simple mechanism for inducing functional modularity in standard architectures.