Never mind the metrics—what about the uncertainty? Visualising binary confusion matrix metric distributions to put performance in perspective
Jul 24, 2023
Speakers
David Lovell
Speaker · 0 followers
Dimity Miller
Speaker · 0 followers
Jaiden Capra
Speaker · 0 followers
Andrew Bradley
Speaker · 0 followers
About
There are strong incentives to build classification systems that show outstanding performance on various datasets and benchmarks. This can encourage a narrow focus on models and the performance metrics used to evaluate and compare them mdash;resulting in a growing body of literature to evaluate and compare _metrics_. This paper strives for a more balanced perspective on binary classifier performance metrics by showing how uncertainty in these metrics can easily eclipse differences in empirical performance. We emphasise the discrete nature of confusion matrices and show how they can be well represented in a 3D lattice whose cross-sections form the space of receiver operating characteristic (ROC) curves. We develop novel interactive visualisations of performance metric contours within (and beyond) ROC space, showing the discrete probability mass functions of true and false positive rates and how these relate to performance metric distributions. We aim to raise awareness of the substantial uncertainty in performance metric estimates that can arise when classifiers are evaluated on empirical datasets and benchmarks, and that performance claims should be tempered by this understanding.There are strong incentives to build classification systems that show outstanding performance on various datasets and benchmarks. This can encourage a narrow focus on models and the performance metrics used to evaluate and compare them mdash;resulting in a growing body of literature to evaluate and compare _metrics_. This paper strives for a more balanced perspective on binary classifier performance metrics by showing how uncertainty in these metrics can easily eclipse differences in empirical p…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
An Effective Meaningful Way to Evaluate Survival Models
Watch later
Shi-ang Qi, …
Distribution-dependent McDiarmid-type Inequalities for Functions of Unbounded Interaction
Watch later
Shaojie Li, …
Platinum Sponsors Panel
Watch later
Exploring Model Dynamics for Accumulative Poisoning Discovery
Watch later
Jianing Zhu, …
Multi-Task Differential Privacy Under Distribution Skew
Watch later
