Search papers, labs, and topics across Lattice.
This paper introduces Pre-Warm, a zero-training-cost method for initializing the first convolutional layer of neural networks based on data-conditioned local patches. By employing MiniBatchKMeans for clustering and inverse Manhattan spatial weighting, Pre-Warm initializes half of the filters with centroids derived from a single training batch, leading to statistically significant accuracy improvements across five standard benchmarks. The method not only enhances performance over traditional Kaiming initialization but also integrates seamlessly into existing training pipelines with minimal overhead.
A lightweight, input-conditioned initialization method boosts CNN performance without additional training costs, achieving significant accuracy gains across multiple datasets.
We introduce Pre-Warm, a simple yet effective zero-training-cost method for data-conditioned initialization of the first convolutional layer. Before the first forward pass, Pre-Warm extracts mean-centered local patches from a single training batch, clusters them with MiniBatchKMeans, applies inverse Manhattan spatial weighting, and uses the resulting centroids to initialize half of the first-layer filters (the remainder retain Kaiming initialization). We derive closed-form rules for all hyperparameters except a single insensitive scale parameter, though we derive a Kaiming parity bound on scale from patch dimensionality. For grayscale datasets we use Otsu's foreground density; for natural color images we use the mean L2 norm of mean-centered patches. Both rules accurately predict the optimal patch count observed in grid search. Across five standard benchmarks -- MNIST, Fashion-MNIST, CIFAR-10, SVHN, and CIFAR-100 -- and 8-seed paired experiments, Pre-Warm yields statistically significant accuracy improvements over standard Kaiming initialization (p < 0.05 on all datasets, p = 0.0007 on SVHN with 8/8 wins, p = 0.0033 on CIFAR-100 with 7/8 wins). The method adds negligible overhead, requires no architectural changes, and integrates into existing training pipelines with only a few lines of code. Pre-Warm demonstrates that even a lightweight, input-dependent signal can meaningfully improve optimization trajectories in modern convolutional networks.
