Search papers, labs, and topics across Lattice.
This paper introduces the concept of split-candidate scaling as a capacity parameter for gradient boosting decision trees (GBDTs), addressing the lack of a single-axis capacity parameter analogous to those in neural networks. By increasing the split-candidate budget, the authors refine the feature-quantization grid and expand the dictionary of paths for boosting updates, leading to a double descent phenomenon where test error initially peaks before decreasing. Empirical analyses confirm that deeper trees and larger training sets necessitate finer grids, highlighting the intricate relationship between candidate-induced geometry and boosting dynamics in GBDTs.
Test error in gradient boosting decision trees can peak and then drop as the split-candidate budget increases, revealing a surprising double descent effect tied to the geometry of candidate paths.
Double descent is commonly studied by scaling an explicit capacity parameter, such as neural-network width. For gradient boosting decision trees (GBDTs), however, an analogous single-axis capacity parameter has not been established. We propose the number of split candidates as an operational capacity parameter for GBDTs. Holding other training controls fixed, increasing the split-candidate budget refines the feature-quantization grid and expands the dictionary of root-to-leaf paths from which boosting selects its updates. To analyze this expansion, we construct an empirical tree-kernel diagnostic that summarizes how candidate-induced paths group the training examples. A regime in which the empirical kernel rank grows toward the sample size and very small positive eigenvalues emerge exposes noise-sensitive directions; in this regime, test error peaks before decreasing again at larger split-candidate budgets. This perspective predicts that deeper trees should reach the regime with fewer split candidates, larger training sets should require finer grids, and label noise should make the peak more pronounced. Experiments support these predictions and show test-error peaks at intermediate split-candidate budgets across XGBoost, LightGBM, and CatBoost, whereas a random-forest control improves monotonically under the same split-candidate sweep. Taken together, our analysis and experiments support split-candidate scaling as a single-axis capacity intervention for studying GBDTs and suggest that the observed double descent arises from an interaction between candidate-induced geometry and boosting dynamics.
