Search papers, labs, and topics across Lattice.
The paper introduces CASHomon sets, which are Rashomon sets extended to the combined algorithm selection and hyperparameter optimization (CASH) setting, allowing for the exploration of well-performing models across multiple model classes. They propose TruVaRImp, a model-based active learning algorithm for efficient CASHomon set identification with theoretical convergence guarantees. Experiments on synthetic and real-world datasets demonstrate that TruVaRImp reliably identifies CASHomon sets and outperforms various baselines, highlighting the importance of considering multiple model classes for data interpretation.
Questioning the common practice of interpreting data through a single model class, this work reveals the existence of alternative well-performing models across multiple model classes and their hyperparameters.
Rashomon sets are model sets within one model class that perform nearly as well as a reference model from the same model class. They reveal the existence of alternative well-performing models, which may support different interpretations. This enables selecting models that match domain knowledge, hidden constraints, or user preferences. However, efficient construction methods currently exist for only a few model classes. Applied machine learning usually searches many model classes, and the best class is unknown beforehand. We therefore study Rashomon sets in the combined algorithm selection and hyperparameter optimization (CASH) setting and call them CASHomon sets. We propose TruVaRImp, a model-based active learning algorithm for level set estimation with an implicit threshold, and provide convergence guarantees. On synthetic and real-world datasets, TruVaRImp reliably identifies CASHomon sets members and matches or outperforms naive sampling, Bayesian optimization, classical and implicit level set estimation methods, and other baselines. Our analyses of predictive multiplicity and feature-importance variability across model classes question the common practice of interpreting data through a single model class.
