Search papers, labs, and topics across Lattice.
The paper introduces TESLA, a novel activation function that combines sine and cosine terms to effectively tackle the parity problem, which is challenging for standard neural networks due to its linear inseparability. By constraining the coefficients of TESLA, the authors establish theoretical bounds on Lipschitz and Rademacher complexity, enhancing training dynamics to prioritize higher-frequency components. Empirical results demonstrate that TESLA achieves strong generalization on the parity task with minimal training data and maintains robustness against significant label noise, while also performing competitively on ImageNet-100, suggesting its broader applicability in vision tasks.
TESLA not only solves the parity problem with minimal data but also excels in robustness, outperforming traditional methods under label noise.
The parity problem--deciding whether the number of ones in a binary vector is odd or even--remains challenging for standard neural networks due to linear inseparability and the need for global interactions. We propose TESLA, an activation defined as a learnable combination of sine and cosine terms, enabling explicit control over polynomial degree and selective amplification of high-order components. Theoretically, we show that constraining TESLA's coefficients yields Lipschitz/Rademacher complexity bounds and shapes the training dynamics to emphasize higher-frequency structure. Empirically, on parity with input length n = 32, TESLA attains strong generalization with 100K training samples (approximately 0.002% of the 2^32 input space) and remains robust under heavy corruption, retaining high accuracy with up to 30% label noise. We also compare against periodic and frequency-based baselines (SIREN, SNAKE, and Fourier feature embeddings) on parity and Forrelation. Beyond synthetic structure, TESLA delivers comparable performance on ImageNet-100, indicating that activation-level degree control transfers to more general vision workloads. Code: https://github.com/KAU-QuantumAILab/TESLA
