Search papers, labs, and topics across Lattice.
This paper investigates feature bagging, an ensemble method that enhances algorithmic stability by aggregating base learners trained on randomly subsampled feature subsets. The authors introduce a novel metric called feature instability (FI), which quantifies the sensitivity of model performance to the removal of individual features, revealing its complementary role to instance instability (II). Experimental results demonstrate that feature bagging significantly improves stability in both parametric linear models and model-free settings, with formal guarantees indicating that even a few rounds of bagging can yield near-optimal stability levels.
Feature bagging can dramatically enhance model stability, achieving near-optimal performance with just a few rounds of aggregation.
We study feature bagging through the lens of algorithmic stability. Feature bagging is an ensemble strategy that aggregates base learners trained on randomly subsampled feature subsets, possibly in a data-dependent manner. We introduce feature instability (FI), the feature-axis analogue of instance instability (II), which measures sensitivity to removing a single feature. Smaller values of II or FI correspond to stronger stability, and our experiments show that FI captures generalization-relevant information complementary to II. Within this framework, we analyze feature bagging in both a parametric linear model and a model-free setting inspired by recursive feature subsampling in random forests. In both settings, we establish formal guarantees showing that feature bagging improves the relevant stability relative to its non-bagged counterpart, with larger improvements under more aggressive subsampling. We further show that a modest number of bagging rounds is sufficient to approach the infinite-bagging stability level.
