Deep RL at the Edge of the Statistical Precipice
Dec 6, 2021
Speakers
Rishabh Agarwal
Speaker · 2 followers
Max Schwarzer
Speaker · 1 follower
Pablo Samuel Castro
Speaker · 1 follower
Aaron Courville
Speaker · 3 followers
Marc Bellemare
Speaker · 3 followers
About
Deep reinforcement learning (RL) algorithms are predominantly evaluated by comparing their relative performance on a large suite of tasks. Most published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks. However, only reporting point estimates ignores the statistical uncertainty implied by the use of a finite number of evaluation runs. Beginning with the Arcade Learning Environment (ALE), the shift towards computationally-demanding benchmarks has led to the practice of evaluating only a handful of runs per task, exacerbating the statistical uncertainty in point estimates. In this paper, we argue that reliable evaluation in the few-run deep RL regime cannot ignore the uncertainty in results without running the risk of slowing down progress in the field. We illustrate this point using a case study on the Atari 100k benchmark, where we find substantial discrepancies between conclusions drawn from point estimates alone versus a more thorough statistical analysis. With the aim of increasing the field's confidence in reported results with a handful of runs, we assert reporting interval estimates of aggregate performance and propose performance distributions to account for the variability in results, as well as present more robust and efficient aggregate metrics, such as interquartile mean scores, to achieve small uncertainty in results. Using such statistical tools, we scrutinize performance evaluations of existing algorithms on other widely used benchmarks including the ALE, Procgen, and the DeepMind Control Suite, again revealing discrepancies in prior comparisons. Our findings call for a change in how we evaluate performance in deep RL, for which we present a more rigorous evaluation methodology to prevent unreliable results from stagnating the field.Deep reinforcement learning (RL) algorithms are predominantly evaluated by comparing their relative performance on a large suite of tasks. Most published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks. However, only reporting point estimates ignores the statistical uncertainty implied by the use of a finite number of evaluation runs. Beginning with the Arcade Learning Environment (ALE), the shift towards computationally-…
Organizer
NeurIPS 2021
Account · 1.9k followers
Categories
AI & Data Science
Category · 10.8k presentations
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Modeling other Agents
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Integrating single-cell multi-omic datasets with optimal transport
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Kernel Identification Through Transformers
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning Optimal Predictive Checklists
Watch later
Haoran Zhang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning Physics Models that Generalize
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Habitat 2.0: Training Home Assistants to Rearrange their Habitat
Watch later
Andrew Szot, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%