The Neural Testbed: Evaluating Joint Predictions
Nov 28, 2022
Speakers
About
Predictive distributions quantify uncertainties ignored by point estimates. This paper introduces The Neural Testbed: an open source benchmark for controlled and principled evaluation of agents that generate such predictions. Crucially, the testbed assesses agents not only on the quality of their marginal predictions per input, but also on their joint predictions across many inputs. We evaluate a range of agents using a simple neural network data generating process.Our results indicate that some popular Bayesian deep learning agents do not fare well with joint predictions, even when they can produce accurate marginal predictions. We also show that the quality of joint predictions drives performance in downstream decision tasks. We find these results are robust across choice a wide range of generative models, and highlight the practical importance of joint predictions to the community.Predictive distributions quantify uncertainties ignored by point estimates. This paper introduces The Neural Testbed: an open source benchmark for controlled and principled evaluation of agents that generate such predictions. Crucially, the testbed assesses agents not only on the quality of their marginal predictions per input, but also on their joint predictions across many inputs. We evaluate a range of agents using a simple neural network data generating process.Our results indicate that some…
Organizer
NeurIPS 2022
Account · 954 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
Watch later
Hung Le, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Machine Learning for Climate Risk
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Towards Better Evaluation for Dynamic Link Prediction
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Test-Time Training with Masked Autoencoders
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Robust Rent Division
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%