Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper analyzes the surprisingly high accuracies of ML models in spectroscopy, arguing that the high dimensionality of spectral data allows models to achieve near-perfect accuracy even when chemical distinctions are absent. They use the Feldman-Hajek theorem and concentration of measure to show that infinitesimal distributional differences become perfectly separable in high-dimensional spaces. Experiments on synthetic and real fluorescence spectra confirm that models can achieve high accuracy using spectrally irrelevant features, leading to misleading interpretations.
Seemingly accurate ML models for spectroscopy may be learning to classify noise and artifacts rather than chemically meaningful features, thanks to the curse of dimensionality.
Machine learning (ML) models have achieved strikingly high accuracies in spectroscopic classification tasks, often without a clear proof that those models used chemically meaningful features. Existing studies have linked these results to data preprocessing choices, noise sensitivity, and model complexity, but no unifying explanation is available so far. In this work, we show that these phenomena arise naturally from the intrinsic high dimensionality of spectral data. Using a theoretical analysis grounded in the Feldman-Hajek theorem and the concentration of measure, we show that even infinitesimal distributional differences, caused by noise, normalisation, or instrumental artefacts, may become perfectly separable in high-dimensional spaces. Through a series of specific experiments on synthetic and real fluorescence spectra, we illustrate how models can achieve near-perfect accuracy even when chemical distinctions are absent, and why feature-importance maps may highlight spectrally irrelevant regions. We provide a rigorous theoretical framework, confirm the effect experimentally, and conclude with practical recommendations for building and interpreting ML models in spectroscopy.