Search papers, labs, and topics across Lattice.
This paper extends the Mixture of Decision Trees (MoDT) framework by integrating heterogeneous expert families, including decision trees, linear support vector machines, and quadratic discriminant analysis, under a unified probabilistic gating mechanism. This approach enhances interpretability and allows for adaptive specialization of experts based on local data geometry, while maintaining coherent likelihood-based inference through calibration of non-probabilistic experts. Experimental results on various datasets show that this heterogeneous mixture-of-experts framework achieves competitive predictive performance compared to traditional homogeneous models and Random Forests, while providing clearer expert assignments.
Heterogeneous expert families can significantly boost interpretability and predictive performance in machine learning models, adapting to local data structures more effectively than traditional homogeneous approaches.
Mixture-of-Experts (MoE) models provide a flexible framework for partitioning complex prediction problems into simpler local learning tasks through an input-dependent gating mechanism. Existing interpretable MoE approaches, such as Mixture of Decision Trees (MoDT), achieve transparency by employing homogeneous decision-tree experts, but this restricts the model to a single inductive bias across all regions of the feature space. We extend the MoDT framework by introducing heterogeneous expert families comprising decision trees, linear support vector machines, and quadratic discriminant analysis under a common probabilistic gating mechanism. To ensure coherent likelihood-based inference, non-probabilistic experts are calibrated to produce conditional class probabilities, allowing parameter estimation within the generalized Expectation-Maximization framework of MoDT. We further establish theoretical monotone ascent guarantees for the proposed heterogeneous gating updates, providing a justification for the optimization procedure. Experiments on a diverse collection of synthetic and real-world benchmark datasets demonstrate that the proposed framework adaptively specializes experts according to local data geometry, yielding interpretable expert assignments while achieving predictive performance competitive with homogeneous MoDT and Random Forests. The proposed approach combines interpretability, adaptive inductive bias selection, and probabilistic coherence within a unified mixture-of-experts framework.
