Why So Pessimistic? Estimating uncertainties for offline RL through Ensembles, and why their independence matters
Nov 28, 2022
Speakers
Kamyar Ghasemipour
Speaker · 0 followers
Shixiang Gu
Speaker · 0 followers
Ofir Nachum
Speaker · 2 followers
About
Motivated by the success of ensembles for uncertainty estimation in supervised learning, we take a renewed look at how ensembles of Q-functions can be leveraged as the primary source of pessimism for offline reinforcement learning (RL). We begin by identifying a critical flaw in a popular algorithmic choice used by many ensemble-based RL algorithms, namely the use of shared pessimistic target values when computing each ensemble member’s Bellman error. Through theoretical analyses and construction of examples in toy MDPs, we demonstrate that shared pessimistic targets can paradoxically lead to value estimates that are effectively optimistic. Given this result, we propose MSG, a practical offline RL algorithm that trains an ensemble of Q-functions with independently computed targets based on completely separate networks, and optimizes a policy with respect to the lower confidence bound of predicted action values. Our experiments on the popular D4RL and RL Unplugged offline RL benchmarks demonstrate that on challenging domains such as antmazes, MSG with deep ensembles surpasses highly well-tuned state-of-the-art methods by a wide margin. Additionally, through ablations on benchmarks domains, we verify the critical significance of using independently trained Q-functions, and study the role of ensemble size. Finally, as MSG’s use of separate networks per ensemble member can become computationally costly with larger neural network architectures, we investigate whether efficient ensemble approximations developed for supervised learning can be similarly effective, and demonstrate that they do not match the performance and robustness of MSG, highlighting the need for new efforts into efficient uncertainty estimation directed at RL.Motivated by the success of ensembles for uncertainty estimation in supervised learning, we take a renewed look at how ensembles of Q-functions can be leveraged as the primary source of pessimism for offline reinforcement learning (RL). We begin by identifying a critical flaw in a popular algorithmic choice used by many ensemble-based RL algorithms, namely the use of shared pessimistic target values when computing each ensemble member’s Bellman error. Through theoretical analyses and constructio…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Interaction-Grounded Learning with Action-inclusive Feedback
Watch later
Tengyang Xie, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning to Sample and Aggregate: Few-shot Reasoning over Temporal Knowledge Graph
Watch later
Ruijie Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Panel: Faculty and Queerness
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Theory and Algorithm for Batch Distribution Drift Problems
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Guiding Exploration Towards Impactful Actions
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
SageMix: Saliency-Guided Mixup for Point Clouds
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%