EnvPool
Nov 28, 2022
Speakers
Jiayi Weng
Speaker · 0 followers
About
There has been significant progress in developing reinforcement learning (RL) training systems. Past works such as IMPALA, Apex, Seed RL, Sample Factory, and others aim to improve the system's overall throughput. In this paper, we try to address a common bottleneck in the RL training system, i.e., parallel environment execution, which is often the slowest part of the whole system but receives little attention. With a curated design for paralleling RL environments, we have improved the RL environment simulation speed across different hardware setups, ranging from a laptop, and a modest workstation, to a high-end machine like NVIDIA DGX-A100. On a high-end machine, EnvPool achieves 1 million frames per second for the environment execution on Atari environments and 3 million frames per second on MuJoCo environments. When running on a laptop, the speed of EnvPool is 2.8 times of the Python subprocess. Moreover, great compatibility with existing RL training libraries has been demonstrated in the open-sourced community, including CleanRL, rl_games, DeepMind Acme, etc. Finally, EnvPool allows researchers to iterate their ideas at a much faster pace and has the great potential to become the de facto RL environment execution engine. Example runs show that it takes only 5 minutes to train Atari Pong and MuJoCo Ant, both on a laptop. Envpool has already been open-sourced at https://github.com/sail-sg/envpool.There has been significant progress in developing reinforcement learning (RL) training systems. Past works such as IMPALA, Apex, Seed RL, Sample Factory, and others aim to improve the system's overall throughput. In this paper, we try to address a common bottleneck in the RL training system, i.e., parallel environment execution, which is often the slowest part of the whole system but receives little attention. With a curated design for paralleling RL environments, we have improved the RL enviro…
Organizer
NeurIPS 2022
Account · 957 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Learning Robust Rule Representations for Abstract Reasoning via Internal Inferences
Watch later
Wenbo Zhang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Wasserstein K-means for clustering probability distributions
Watch later
Yubo Zhuang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Discussion
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning Multi-resolution Functional Maps with Spectral Attention for Robust Shape Matching
Watch later
Lei Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Follow-the-Perturbed-Leader for Adversarial Markov Decision Processes with Bandit Feedback
Watch later
Yan Dai, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Old can be Gold: Better Gradient Flow can make Vanilla-GCNs Great Again
Watch later
Ajay Jaiswal, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%