Search papers, labs, and topics across Lattice.
The paper introduces Composite Silhouette, a novel internal criterion for determining the optimal number of clusters in unsupervised learning, addressing the biases of micro- and macro-averaged Silhouette scores. It combines micro- and macro-averaged Silhouette scores on repeated subsamples using an adaptive convex weight determined by their normalized discrepancy. Theoretical analysis provides finite-sample concentration guarantees, and empirical results demonstrate improved accuracy in recovering the true number of clusters compared to standard Silhouette variants.
Subsampling and adaptively weighting micro- and macro-averaged Silhouette scores can more accurately determine the number of clusters than relying on a single partition.
Determining the number of clusters is a central challenge in unsupervised learning, where ground-truth labels are unavailable. The Silhouette coefficient is a widely used internal validation metric for this task, yet its standard micro-averaged form tends to favor larger clusters under size imbalance. Macro-averaging mitigates this bias by weighting clusters equally, but may overemphasize noise from under-represented groups. We introduce Composite Silhouette, an internal criterion for cluster-count selection that aggregates evidence across repeated subsampled clusterings rather than relying on a single partition. For each subsample, micro- and macro-averaged Silhouette scores are combined through an adaptive convex weight determined by their normalized discrepancy and smoothed by a bounded nonlinearity; the final score is then obtained by averaging these subsample-level composites. We establish key properties of the criterion and derive finite-sample concentration guarantees for its subsampling estimate. Experiments on synthetic and real-world datasets show that Composite Silhouette effectively reconciles the strengths of micro- and macro-averaging, yielding more accurate recovery of the ground-truth number of clusters.
