Why So Pessimistic? Estimating uncertainties for offline RL through Ensembles, and why their independence matters
Nov 28, 2022
Speakers
Kamyar Ghasemipour
Speaker · 0 followers
Shixiang Gu
Speaker · 0 followers
Ofir Nachum
Speaker · 2 followers
About
Motivated by the success of ensembles for uncertainty estimation in supervised learning, we take a renewed look at how ensembles of Q-functions can be leveraged as the primary source of pessimism for offline reinforcement learning (RL). We begin by identifying a critical flaw in a popular algorithmic choice used by many ensemble-based RL algorithms, namely the use of shared pessimistic target values when computing each ensemble member’s Bellman error. Through theoretical analyses and construction of examples in toy MDPs, we demonstrate that shared pessimistic targets can paradoxically lead to value estimates that are effectively optimistic. Given this result, we propose MSG, a practical offline RL algorithm that trains an ensemble of Q-functions with independently computed targets based on completely separate networks, and optimizes a policy with respect to the lower confidence bound of predicted action values. Our experiments on the popular D4RL and RL Unplugged offline RL benchmarks demonstrate that on challenging domains such as antmazes, MSG with deep ensembles surpasses highly well-tuned state-of-the-art methods by a wide margin. Additionally, through ablations on benchmarks domains, we verify the critical significance of using independently trained Q-functions, and study the role of ensemble size. Finally, as MSG’s use of separate networks per ensemble member can become computationally costly with larger neural network architectures, we investigate whether efficient ensemble approximations developed for supervised learning can be similarly effective, and demonstrate that they do not match the performance and robustness of MSG, highlighting the need for new efforts into efficient uncertainty estimation directed at RL.Motivated by the success of ensembles for uncertainty estimation in supervised learning, we take a renewed look at how ensembles of Q-functions can be leveraged as the primary source of pessimism for offline reinforcement learning (RL). We begin by identifying a critical flaw in a popular algorithmic choice used by many ensemble-based RL algorithms, namely the use of shared pessimistic target values when computing each ensemble member’s Bellman error. Through theoretical analyses and constructio…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
A Future for AI Governance Systems beyond Predictions
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Robust Models are less Over-Confident
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Hybrid RL: Using Both Offline and Online Data can Make RL Efficient
Watch later
Yuda Song, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
SurDis: A Surface Discontinuity Dataset for Wearable Technology to Assist Blind Navigation in Urban Environments
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Addressing Bias in Face Detectors using Decentralised Data collection with incentives
Watch later
Ahan M R, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Can Querying for Bias Leak Protected Attributes? Achieving Privacy With Smooth Sensitivity
Watch later
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%