Search papers, labs, and topics across Lattice.
The paper introduces Decoupled Descent (DD), a training algorithm designed to enforce train error to asymptotically track test error in stylized Gaussian mixture models, addressing the generalization gap. DD leverages approximate message passing theory to iteratively cancel biases from data reuse, achieving zero-cost validation and 100% data utilization within this regime. Experiments on XOR classification, noisy MNIST, and CIFAR-10 probing show that DD narrows the generalization gap compared to gradient descent, even when assumptions are relaxed.
Achieve perfect train-test error tracking with a new training algorithm, Decoupled Descent, that eliminates the need for validation sets in certain stylized settings.
In modern parametric model training, full-batch gradient descent (and its variants) suffers due to progressively stronger biasing towards the exact realization of training data; this drives the systematic ``generalization gap'', where the train error becomes an unreliable proxy for test error. Existing approaches either argue this gap is benign through complex analysis or sacrifice data to a validation set. In contrast, we introduce decoupled descent (DD), a novel theory-based training algorithm that satisfies a train-test identity -- enforcing the train error to asymptotically track the test error for stylized Gaussian mixture models. Within this specific regime, leveraging approximate message passing theory, DD iteratively cancels the biases due to data reuse, rigorously demonstrating the feasibility of zero-cost validation and $100\%$ data utilization. Moreover, DD is governed by a low-dimensional state evolution recursion, rendering the dynamics of the algorithm transparent and tractable. We validate DD on XOR classification, yielding superior performance compared to GD; additionally, we implement noisy MNIST and non-linear probing of CIFAR-10, demonstrating that even when our stylized assumptions are relaxed, DD narrows the generalization gap compared to GD.
