Supervised Q-Learning for Continuous Control
Dec 2, 2022
Speakers
Taiyi Wang
Speaker · 0 followers
Bolei Zhou
Speaker · 0 followers
About
Policy gradient (PG) algorithms have been widely used in reinforcement learning (RL). However, PG algorithms rely on exploiting the value function being learned with the first-order update locally, which results in limited sample efficiency. In this work, we propose an alternative method called Zeroth-Order Supervised Policy Improvement (ZOSPI). ZOSPI exploits the estimated value function Q globally while preserving the local exploitation of the PG methods based on zeroth-order policy optimization. This learning paradigm follows Q-learning but overcomes the difficulty of efficiently operating argmax in continuous action space. It finds max-valued action within a small number of samples. The policy learning of ZOSPI has two steps: First, it samples actions and evaluates those actions with a learned value estimator, and then it learns to perform the action with the highest value through supervised learning. We further demonstrate such a supervised learning framework can learn multi-modal policies. Experiments show that ZOSPI achieves competitive results on the continuous control benchmarks with a remarkable sample efficiency.Policy gradient (PG) algorithms have been widely used in reinforcement learning (RL). However, PG algorithms rely on exploiting the value function being learned with the first-order update locally, which results in limited sample efficiency. In this work, we propose an alternative method called Zeroth-Order Supervised Policy Improvement (ZOSPI). ZOSPI exploits the estimated value function Q globally while preserving the local exploitation of the PG methods based on zeroth-order policy optimizati…
Organizer
NeurIPS 2022
Account · 961 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Attracting and Dispersing: A Simple Approach for Source-free Domain Adaptation
Watch later
Shiqi Yang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Cooperation or Competition: Avoiding Player Domination for Multi-target Robustness by Adaptive Budgets
Watch later
Yimu Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The Dollar Street Dataset: Images Representing the Geographic and Socioeconomic Diversity of the World
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Generative Collage and its Sticky Questions on Human-AI Co-Creativity
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
MVP: Practical Adversarial Multivalid Conformal Prediction
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Dual-Generator Offlien Reinforcement Learning
Watch later
Quan Vuong, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%