Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of the Derivative Regularization penalty (DREG) across a comprehensive set of 960 experiments involving various activations, regularizers, datasets, and random seeds. The results reveal that DREG outperforms other regularizers in terms of overall accuracy and noise robustness, particularly excelling under the GELU activation function commonly used in transformer architectures. Notably, DREG's performance is especially pronounced in scenarios with limited data, highlighting its potential as a geometric inductive bias that compensates for data scarcity.
DREG not only achieves superior accuracy but also thrives in data-scarce environments, making it a game-changer for deep learning regularization strategies.
We present a large-scale empirical study isolating the contributions of the Derivative Regularization penalty (DREG). Across a fully-crossed factorial sweep of 960 experiments spanning 4 activations, 6 regularizers, 8 datasets, and 5 random seeds, we ask: when, where, and why does DREG work? Our results establish three principal findings. First, DREG achieves the highest overall and clean-regime accuracy among all regularizers evaluated (significantly so against the unregularized baseline, Weight Decay, and IGPen; Wilcoxon $p \leq 0.031$). It ranks second in noise robustness behind Spectral Normalization (SN) - the only two layer-wise regularizers in the study. Second, DREG is globally the best-performing regularizer under GELU, the default activation in modern transformer architectures, particularly on both messy vision and messy NLP benchmarks, suggesting direct applicability to frontier deep learning settings. Third, DREG's advantage over competing regularizers is most pronounced under data scarcity, consistent with its role as a geometric inductive bias that substitutes for the regularizing effect of data volume. Throughout, DREG is applied with a single fixed hyperparameter $位= 10^{-2.5}$ and no per-dataset tuning, supporting its characterization as a plug-and-play regularizer for neural networks with nontrivial Jacobian structure. These findings are consistent with DREG's design: concentrating regularization pressure on layers where the activation derivative is largest, rather than constraining the network uniformly.
