Search papers, labs, and topics across Lattice.
This paper introduces a novel "verify-through-training" paradigm for neural networks, where the goal is to design networks that are easily verifiable using the computationally efficient "trivial bound" of Lipschitz constants. They identify and mitigate structural obstructions to a tight trivial bound, such as dead neurons and ill-conditioned weights, through architectural innovations like norm-saturating polyactivations and bias-free sinusoidal layers. The result is robust networks with provably small and tight Lipschitz bounds, achieved without the computational overhead of complex verification techniques.
Forget expensive verification: training networks to be *trivially* verifiable yields state-of-the-art Lipschitz bounds and adversarial robustness.
The global Lipschitz constant of a neural network governs both adversarial robustness and generalization. Conventional approaches to ``certified training" typically follow a train-then-verify paradigm: they train a network and then attempt to bound its Lipschitz constant. Because the efficient ``trivial bound" (the product of the layerwise Lipschitz constants) is exponentially loose for arbitrary networks, these approaches must rely on computationally expensive techniques such as semidefinite programming, mixed-integer programming, or branch-and-bound. We propose a different paradigm: rather than designing complex verifiers for arbitrary networks, we design networks to be verifiable by the fast trivial bound. We show that directly penalizing the trivial bound during training forces it to become tight, thereby effectively regularizing the true Lipschitz constant. To achieve this, we identify three structural obstructions to a tight trivial bound (dead neurons, bias terms, and ill-conditioned weights) and introduce architectural mitigations, including a novel notion of norm-saturating polyactivations and bias-free sinusoidal layers. Our approach avoids the runtime complexity of advanced verification while achieving strong results: we train robust networks on MNIST with Lipschitz bounds that are small (orders of magnitude lower than comparable works) and tight (within 10% of the ground truth). The experimental results validate the theoretical guarantees, support the proposed mechanisms, and extend empirically to diverse activations and non-Euclidean norms.
