Search papers, labs, and topics across Lattice.
This paper investigates the use of normalized generalized means of order *r* for density aggregation, a key task in combining predictions from deep ensembles. By analyzing the log-likelihood of these means, the authors identify optimal configurations for different scenarios and demonstrate that only the range *r* ∈ [0,1] guarantees systematic improvements over individual distributions. The findings provide a theoretical justification for the widespread use of linear and geometric pooling while highlighting the potential failure of aggregation rules outside this range, validated through experiments on image and text classification.
Linear and geometric pooling aren't just common ensemble methods, they're theoretically optimal for consistent gains, while other aggregation rules can actively hurt performance.
Density aggregation is a central problem in machine learning, for instance when combining predictions from a Deep Ensemble. The choice of aggregation remains an open question with two commonly proposed approaches being linear pooling (probability averaging) and geometric pooling (logit averaging). In this work, we address this question by studying the normalized generalized mean of order $r \in \mathbb{R} \cup \{-\infty,+\infty\}$ through the lens of log-likelihood, the standard evaluation criterion in machine learning. This provides a unifying aggregation formalism and shows different optimal configurations for different situations. We show that the regime $r \in [0,1]$ is the only range ensuring systematic improvements relative to individual distributions, thereby providing a principled justification for the reliability and widespread practical use of linear ($r=1$) and geometric ($r=0$) pooling. In contrast, we show that aggregation rules with $r \notin [0,1]$ may fail to provide consistent gains with explicit counterexamples. Finally, we corroborate our theoretical findings with empirical evaluations using Deep Ensembles on image and text classification benchmarks.
