Bounding the fairness and accuracy of classifiers from population statistics

Jul 12, 2020



Machine Learning models increasingly affect people's lives, necessitating their audit for fairness and accuracy when applied to diverse populations. However, these models are often difficult to asses, either because they are considered proprietary by their developers in industry or government, or because no individual-level validation data is available to assess them. We consider a classification model applied to a given task, whose properties are impossible to estimate using a validation set, either due to an absence of representative individual-level labeled data or because access to the classifier, even as a black-box model, is impossible. Instead, only aggregate statistics on the rate of positive predictions in each of several sub-populations are available. In addition, we have access to the true rates of positive labels in each of these sub-populations. We show that these aggregate statistics can be used to lower-bound the discrepancy of a classifier, which is a measure that balances inaccuracy and unfairness. To this end, we define a new measure of unfairness, which is equal to the fraction of the population on which the classifier misbehaves, compared to its global, ideally fair behavior, as defined by the measure of equalized odds. We propose an efficient and practical procedure for finding the best possible lower bound on the discrepancy of the classifier, given the aggregate statistics. Experiments demonstrate the empirical tightness of this lower bound, as well as its possible uses on various types of problems, ranging from estimating the quality of voting polls to measuring the effectiveness of patient identification from internet search queries.



About ICML 2020

The International Conference on Machine Learning (ICML) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence known as machine learning. ICML is globally renowned for presenting and publishing cutting-edge research on all aspects of machine learning used in closely related areas like artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, and robotics. ICML is one of the fastest growing artificial intelligence conferences in the world. Participants at ICML span a wide range of backgrounds, from academic and industrial researchers, to entrepreneurs and engineers, to graduate students and postdocs.

Store presentation

Should this presentation be stored for 1000 years?

How do we store presentations

Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%


Recommended Videos

Presentations on similar topic, category or speaker