Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This work systematically re-evaluates regression objectives for deep learning on tabular data, finding that while many alternatives offer inconsistent gains, quantile loss consistently outperforms a tuned MSE baseline. However, even quantile loss is limited by the need for different optimal stopping times across regions with varying target noise, which a single global early-stopping checkpoint cannot accommodate. To address this, a heteroskedasticity-aware extension of quantile loss using input-dependent label smoothing is introduced, showing further improvements.
Quantile loss consistently beats MSE on tabular regression, but global early stopping still compromises performance in noisy regions, necessitating input-dependent label smoothing.
While deep learning models for tabular data have recently received significant research attention, the choice of objective functions for regression tasks remains an understudied degree of freedom. Several lines of research offer alternatives to the standard Mean Squared Error (MSE), but differences in datasets, backbones, and evaluation protocols make their practical benefits difficult to compare. In this work, we address this gap by performing a systematic re-evaluation of regression objectives across a broad collection of established tabular datasets and multiple neural backbones. Our comparison reveals that many proposed objectives provide limited or inconsistent gains over a properly tuned MSE baseline, while quantile loss yields consistent aggregate improvements over MSE across the tested backbones. We then identify a limitation that remains even with this strong objective: regions with different levels of target noise can reach their best predictive performance at different training epochs. Consequently, a single global early-stopping checkpoint can impose a compromise between uncertainty regions, even when the objective can represent heteroskedastic target distributions. Motivated by this limitation, we introduce a heteroskedasticity-aware extension of quantile loss that uses input-dependent label smoothing as a potential remedy and improves over quantile loss on several backbones. Overall, our results provide a modern empirical picture of regression objectives for tabular deep learning and highlight how differences in optimal stopping times across uncertainty regions can limit the effectiveness of global early stopping. The code is available at https://github.com/yandex-research/tabular-regression.