The Rashomon Set is All You Need: A Framework for Analyzing Trustworthiness of Trees under Multiplicity
Ethan Hsu, Tony Cao, Lesia Semenova, Chudi Zhong
The Rashomon Effect in machine learning refers to the existence of multiple models that achieve approximately the same predictive performance as the optimal model on a given dataset. Researchers have leveraged this effect for model selection based on desirable properties such as fairness or sparsity. However, a systematic analysis of different properties across models in the Rashomon set (set of near-optimal models) has not been conducted. In this work, we bridge this gap by introducing a framework that evaluates and compares the performance of models from the Rashomon set against models explicitly optimized for a trustworthiness criterion, such as robustness, privacy, or fairness. We focus on decision trees, which are inherently interpretable models. Our results demonstrate that near-optimal decision trees from the Rashomon set can perform comparably to trees explicitly optimized for trustworthiness. This suggests that leveraging the Rashomon set for model selection can be an effective strategy for building reliable machine learning systems. Our framework can be a valuable tool for both benchmarking Rashomon sets of decision trees and studying the trustworthiness properties of interpretable models.