Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Dynamical Batching (DB), a class-cycling training protocol, to investigate its impact on neural network performance and internal structure, drawing inspiration from nonequilibrium systems. DB enables capacity-limited MLPs to achieve perfect accuracy on challenging datasets where Uniform Batching (UB) fails, by facilitating more efficient switching between class-specialized internal states. The study reveals that DB induces a more persistent and localized set of weights responsible for class transitions, a structural signature observed even when accuracy gains are modest.
Class-cycling training can unlock solutions for capacity-limited networks that standard batching misses, by reorganizing internal weight dynamics.
We investigate whether cyclically varying the class composition of training batches can modify both the performance and the internal organization of neural networks. Motivated by nonequilibrium systems, in which cycling between related sub-goals can make switching between their solutions increasingly efficient, we introduce Dynamical Batching (DB), a sampling protocol that periodically emphasizes each class while leaving the loss and the optimizer unchanged, and compare it with Uniform Batching (UB), in which every mini-batch has equal class proportions. In the regime where standard training fails, DB changes the outcome: on a three-arm spiral, a capacity-limited MLP that plateaus far below perfect accuracy under UB fits the training data under DB. Tracking individual weights, we find that under DB the change between class-specialized states is carried by a smaller set of weights than under UB, and that this set is markedly more persistent from cycle to cycle, on every synthetic dataset we test and in a wider network that fits the data under both protocols. This organization is a signature of the protocol rather than a predictor of its benefit: it appears equally where DB leaves accuracy unchanged. On CIFAR-10, the same protocol modestly improves the test accuracy of a channel-narrowed ResNet-10, a standard residual architecture, and of a second capacity-limited convolutional network. These results show that how class emphasis is distributed in time can shape both the solution a network reaches and the internal structure of the network.