Parallel Q-Learning: a Scheme for Time-efficient Reinforcement Learning
Jul 24, 2023
Speakers
Zhang-Wei Hong
Řečník · 0 sledujících
Anurag Ajay
Řečník · 0 sledujících
Pulkit Agrawal
Řečník · 0 sledujících
About
Reinforcement learning algorithms require a long time to learn policies on complex tasks due to the need for a large amount of training data. With the recent advances in GPU-based simulation, such as Isaac Gym, data collection has been sped up thousands of times on a commodity GPU. Most prior works have used on-policy methods such as PPO to train policies due to their simplicity and easy-to-scale nature. Off-policy methods are usually more sample-efficient but more challenging to be scaled up, resulting in a much longer wall-clock training time in practice. In this work, we present a novel Parallel Q-Learning (PQL) scheme that is substantially faster in wall-clock time and achieves better sample efficiency than PPO. The key enabling factor is parallelizing the data collection, policy function learning, and value function learning. Different from prior works on distributed off-policy learning, such as Apex, our scheme is designed specifically for massively parallel GPU-based simulation and optimized to work on a single workstation. We demonstrate the capability of scaling up Q-learning methods to tens of thousands of parallel environments. We also investigate important factors that can affect policy learning speed, including the number of parallel environments, exploration schemes, batch size, GPU models, etc.Reinforcement learning algorithms require a long time to learn policies on complex tasks due to the need for a large amount of training data. With the recent advances in GPU-based simulation, such as Isaac Gym, data collection has been sped up thousands of times on a commodity GPU. Most prior works have used on-policy methods such as PPO to train policies due to their simplicity and easy-to-scale nature. Off-policy methods are usually more sample-efficient but more challenging to be scaled up, r…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts
Zhlédnout později
Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexity
Zhlédnout později
Risheng Liu, …
Interactive Object Placement with Reinforcement Learning
Zhlédnout později
“AI For Good” Isn’t Good Enough: A Call for Human-Centered AI
Zhlédnout později
Achieving Linear Speedup in Non-IID Federated Bilevel Learning
Zhlédnout později
Minhui Huang, …
