Search papers, labs, and topics across Lattice.
This paper introduces Intrinsic Trainability (InTrain), a novel theoretical framework for zero-cost neural architecture search that unifies the concepts of geometric capacity and optimization resilience to assess the trainability of neural networks. By quantifying geometric capacity through the participation ratio of activation covariance eigenspectrum and evaluating optimization resilience via cumulative gradient health, InTrain offers a comprehensive measure of architectural effectiveness. Experimental results show that InTrain achieves ranking correlations comparable to state-of-the-art ensemble-based proxies while outperforming existing single-metric methods on standard NAS benchmarks.
InTrain reveals that the synergy between geometric capacity and optimization resilience is crucial for accurately assessing neural architecture trainability without costly training.
Training-free neural architecture search promises efficient discovery of high-performance networks without costly training. However, existing zero-cost proxies rely on fragmented heuristics that fail to capture the fundamental question: what makes an architecture trainable? This paper introduces Intrinsic Trainability (InTrain), a unified theoretical proxy that formalizes trainability as an architectural invariant emerging from two synergistic components: geometric capacity and optimization resilience. We operationalize intrinsic trainability through analysis of neural information processing. Geometric capacity is quantified via the participation ratio of activation covariance eigenspectrum, capturing the effective dimensionality of representation manifolds. Optimization resilience is measured through cumulative gradient health, assessing the robustness of backpropagation across network depth. InTrain synthesizes these dimensions through a scale-invariant multiplicative coupling, which we hypothesize is essential for capturing their synergistic, non-additive relationship. Extensive experiments on standard NAS benchmarks and search spaces demonstrate that InTrain achieves ranking correlations on par with state-of-the-art ensemble-based proxies and outperforms other single-metric methods.
