Search papers, labs, and topics across Lattice.
The paper introduces ElbowSig, a framework that formalizes the elbow method for selecting the number of clusters by evaluating a normalized discrete curvature statistic against a null distribution of unstructured data. It addresses the limitation of existing methods that focus on a single "optimal" partition by identifying statistically significant structure at multiple resolutions. The authors derive asymptotic properties of the null statistic in large-sample and high-dimensional regimes, and demonstrate that ElbowSig maintains Type-I error control while resolving multiscale organizational structures.
Discover statistically meaningful structure at multiple resolutions in your unsupervised learning tasks, moving beyond single "optimal" partitions with a rigorous, algorithm-agnostic elbow method.
Selecting the number of clusters remains a fundamental challenge in unsupervised learning. Existing criteria typically target a single ``optimal'' partition, often overlooking statistically meaningful structure present at multiple resolutions. We introduce ElbowSig, a framework that formalizes the heuristic ``elbow'' method as a rigorous inferential problem. Our approach centers on a normalized discrete curvature statistic derived from the cluster heterogeneity sequence, which is evaluated against a null distribution of unstructured data. We derive the asymptotic properties of this null statistic in both large-sample and high-dimensional regimes, characterizing its baseline behavior and stochastic variability. As an algorithm-agnostic procedure, ElbowSig requires only the heterogeneity sequence and is compatible with a wide range of clustering methods, including hard, fuzzy, and model-based clustering. Extensive experiments on synthetic and empirical datasets demonstrate that the method maintains appropriate Type-I error control while providing the power to resolve multiscale organizational structures that are typically obscured by single-resolution selection criteria.
