Rashomon Capacity: A Metric for Predictive Multiplicity in Classification
Nov 28, 2022
Speakers
Hsiang Hsu
Speaker · 0 followers
Flavio P. Calmon
Speaker · 1 follower
About
Predictive multiplicity occurs when classification models with nearly indistinguishable average performances assign conflicting predictions to individual samples. When used for decision-making in applications of consequence (e.g., lending, education, criminal justice), models developed without regard for predictive multiplicity may result in unjustified and arbitrary decisions for specific individuals. We introduce a new measure of predictive multiplicity in probabilistic classification called Rashomon capacity. Prior metrics for predictive multiplicity focus on classifiers that output thresholded (i.e., 0-1) predicted classes. In contrast, Rashomon capacity applies to probabilistic classifiers, capturing more nuanced score variations for individual samples. We provide a rigorous derivation for Rashomon capacity, argue its intuitive appeal, and demonstrate how to estimate it in practice. We show that Rashomon capacity yields principled strategies for disclosing conflicting models to stakeholders. Our numerical experiments illustrate how Rashomon capacity captures predictive multiplicity in various datasets and learning models, including neural networks. The tools introduced in this paper can help data scientists measure, report, and ultimately resolve predictive multiplicity prior to model deployment.Predictive multiplicity occurs when classification models with nearly indistinguishable average performances assign conflicting predictions to individual samples. When used for decision-making in applications of consequence (e.g., lending, education, criminal justice), models developed without regard for predictive multiplicity may result in unjustified and arbitrary decisions for specific individuals. We introduce a new measure of predictive multiplicity in probabilistic classification called R…
Organizer
NeurIPS 2022
Account · 953 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Towards Markov Properties for Continuous-Time Dynamical Systems
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Multi-Instance Causal Representation Learning for Instance Label Prediction and Out-of-Distribution Generalization
Watch later
Weijia Zhang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Characterising the Robustness of Reinforcement Learning for Continuous Control using Disturbance Injection
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Weakly Supervised Representation Learning with Sparse Perturbations
Watch later
Kartik Ahuja, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Collaborating with language models for embodied reasoning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning Representations via a Robust Behavioral Metric for Deep Reinforcement Learning
Watch later
Jianda Chen, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%