Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper compares penalized regression, Bayesian, spatial, and tree-based models for predicting provincial poverty in Indonesia, explicitly tackling small sample sizes (n=34) and high collinearity. Using Leave-One-Out Cross-Validation, the study finds that simple linear shrinkage models (Ridge, Elastic Net, LASSO) outperform more complex machine learning ensembles like BART due to overfitting. The analysis identifies ICT skills as a robust predictor of lower provincial poverty across the best-performing models.
Complex machine learning models can be dangerously unreliable in regional datasets with small sample sizes; simple, regularized linear models offer more robust predictions.
Identifying the structural drivers of poverty in regional datasets is frequently hindered by small sample sizes and high multidimensional collinearity, which can result in unstable and misleading policy advice. This paper evaluates the provincial causes of poverty in Indonesia by addressing these specific statistical hazards. We employ a rigorous model-comparison framework designed for small samples ($n=34$) with high collinearity, comparing standard linear models with frequentist penalisation, Bayesian shrinkage priors, an adjusted spatial intrinsic conditionally autoregressive (ICAR) model, and complex machine learning ensembles. To ensure a robust evaluation, we measure predictive performance using strict Leave-One-Out Cross-Validation (LOOCV). The results demonstrate that algorithmic complexity is inherently risky in regional datasets: simple linear shrinkage models (Ridge, Elastic Net, LASSO) achieve the superior out-of-sample prediction, whereas complex ensembles like BART suffer from severe overfitting. Across all successful regularised models, ICT skills consistently emerge as the most stable proxy for lower provincial poverty. The primary contribution of this paper is demonstrating that, in data-constrained regional analysis, parametrically regularised linear shrinkage provides a more reliable mathematical foundation for isolating structural development priorities, such as ICT, than either naive OLS or unconstrained machine learning.