Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SCBoost, a novel boosting framework that replaces traditional residual fitting with residual orthogonalization to reduce learner redundancy. By employing Spectral Residual Projection (SRP) and Covariance-Regularized Weighting (CRW), SCBoost ensures that each learner focuses on capturing new information rather than revisiting correlated errors, leading to improved ensemble performance. Theoretical analysis confirms the benefits of SRP in enhancing the Signal-to-Noise Ratio, and extensive experiments show significant gains in accuracy and F1 score across multiple datasets.
Reducing redundancy in boosting could lead to more efficient ensemble models, with SCBoost achieving superior accuracy and F1 scores by focusing on novel empirical innovations.
While sequential residual fitting is the bedrock of standard boosting frameworks, it inherently breeds learner redundancy by repeatedly revisiting correlated error components. To address this bottleneck, we propose a shift from residual fitting to \textit{residual orthogonalization} and introduce SCBoost. Our framework tackles redundancy through two complementary mechanisms: Spectral Residual Projection (SRP) and Covariance-Regularized Weighting (CRW). During training, SRP projects each residual target onto the orthogonal complement of the historical prediction subspace, forcing successive learners to capture only novel empirical innovations. During aggregation, CRW optimizes ensemble weights on a validation set with an explicit covariance penalty to mitigate remaining correlations. Theoretically, we provide a finite-sample geometric characterization proving that SRP yields an exact additive residual-energy decomposition. Furthermore, under an isotropic-noise assumption, we rigorously establish the conditions under which this projection improves the effective Signal-to-Noise Ratio. Extensive experiments across ten benchmark datasets demonstrate that SCBoost delivers strong out-of-the-box performance, particularly in accuracy and F1 score. This work reinterprets boosting through a geometric lens, suggesting that explicit redundancy control is a principled and necessary step toward more efficient ensemble architectures.