Parallel Q-Learning: a Scheme for Time-efficient Reinforcement Learning
Jul 24, 2023
Speakers
Zhang-Wei Hong
Speaker · 0 followers
Anurag Ajay
Speaker · 0 followers
Pulkit Agrawal
Speaker · 0 followers
About
Reinforcement learning algorithms require a long time to learn policies on complex tasks due to the need for a large amount of training data. With the recent advances in GPU-based simulation, such as Isaac Gym, data collection has been sped up thousands of times on a commodity GPU. Most prior works have used on-policy methods such as PPO to train policies due to their simplicity and easy-to-scale nature. Off-policy methods are usually more sample-efficient but more challenging to be scaled up, resulting in a much longer wall-clock training time in practice. In this work, we present a novel Parallel Q-Learning (PQL) scheme that is substantially faster in wall-clock time and achieves better sample efficiency than PPO. The key enabling factor is parallelizing the data collection, policy function learning, and value function learning. Different from prior works on distributed off-policy learning, such as Apex, our scheme is designed specifically for massively parallel GPU-based simulation and optimized to work on a single workstation. We demonstrate the capability of scaling up Q-learning methods to tens of thousands of parallel environments. We also investigate important factors that can affect policy learning speed, including the number of parallel environments, exploration schemes, batch size, GPU models, etc.Reinforcement learning algorithms require a long time to learn policies on complex tasks due to the need for a large amount of training data. With the recent advances in GPU-based simulation, such as Isaac Gym, data collection has been sped up thousands of times on a commodity GPU. Most prior works have used on-policy methods such as PPO to train policies due to their simplicity and easy-to-scale nature. Off-policy methods are usually more sample-efficient but more challenging to be scaled up, r…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Veridical Data Science with a case study to find genetic drivers of a heart disease
Watch later
Equivariant Architectures for Learning in Deep Weight Spaces
Watch later
Aviv Navon, …
Counterfactuals, play and causal inference in young children and machines
Watch later
Pruning via Sparsity-indexed ODE: A Continuous Sparsity Viewpoint
Watch later
Zhanfeng Mo, …
Conformal Prediction Sets for Graph Neural Networks
Watch later
Dual Focal Loss for Calibration
Watch later
Linwei Tao, …