On All-Action Policy Gradients
Dec 2, 2022
Speakers
Michal Nauman
Speaker · 0 followers
Marek Cygan
Speaker · 0 followers
About
In this paper, we analyze the variance of stochastic policy gradient with many action samples per state (all-action SPG). We decompose the variance of SPG and derive an optimality condition for all-action SPG. The optimality condition shows when all-action SPG should be preferred over single-action counterpart and allows to determine a variance-minimizing sampling scheme in SPG estimation. Furthermore, we propose dynamics-all-action (DAA) module, an augmentation that allows for all-action sampling without manipulation of the environment. DAA addresses the problems associated with using a Q-network for all-action sampling and can be readily applied to any on-policy SPG algorithm. We find that using DAA with a canonical on-policy algorithm (PPO) yields better sample efficiency and higher policy returns on a variety of challenging continuous action environments.In this paper, we analyze the variance of stochastic policy gradient with many action samples per state (all-action SPG). We decompose the variance of SPG and derive an optimality condition for all-action SPG. The optimality condition shows when all-action SPG should be preferred over single-action counterpart and allows to determine a variance-minimizing sampling scheme in SPG estimation. Furthermore, we propose dynamics-all-action (DAA) module, an augmentation that allows for all-action sampli…
Organizer
NeurIPS 2022
Account · 961 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Large Language Models Are Human-Level Prompt Engineers
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Concept Activation Regions: A Generalized Framework For Concept-Based Explanations
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Nonconvex Framework for Structured Dynamic Covariance Recovery
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Opening Remarks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
HyperSound: Generating Implicit Neural Representations of Audio Signals with Hypernetworks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Intelligent transportation systems - ACFR
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%