Truly Deterministic Policy Optimization
Nov 28, 2022
Speakers
Ehsan Saleh
Speaker · 0 followers
Saba Ghaffari
Speaker · 0 followers
Timothy Bretl
Speaker · 0 followers
Matthew West
Speaker · 0 followers
About
In this paper, we present a policy gradient method that avoids exploratory noise injection and performs policy search over the deterministic landscape, with the goal of improving learning with long horizons and non-local rewards. By avoiding noise injection all sources of estimation variance can be eliminated in systems with deterministic dynamics (up to the initial state distribution). Since deterministic policy regularization is impossible using traditional non-metric measures such as the KL divergence, we derive a Wasserstein-based quadratic model for our purposes. We state conditions on the system model under which it is possible to establish a monotonic policy improvement guarantee, propose a surrogate function for policy gradient estimation, and show that it is possible to compute exact advantage estimates if both the state transition model and the policy are deterministic. Finally, we describe two novel robotic control environments—one with non-local rewards in the frequency domain and the other with a long horizon (8000 time-steps)—for which our policy gradient method (TDPO) significantly outperforms existing methods (PPO, TRPO, DDPG, and TD3). Our implementation with all the experimental settings is available at https://anonymous.4open.science/r/code_tdpo-D23A.In this paper, we present a policy gradient method that avoids exploratory noise injection and performs policy search over the deterministic landscape, with the goal of improving learning with long horizons and non-local rewards. By avoiding noise injection all sources of estimation variance can be eliminated in systems with deterministic dynamics (up to the initial state distribution). Since deterministic policy regularization is impossible using traditional non-metric measures such as the KL …
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Online Nonnegative CP-dictionary Learning for Markovian Data
Watch later
Hanbaek Lyu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Membership Inference Attacks via Adversarial Examples
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Free Probability for predicting the performance of feed-forward fully connected neural networks
Watch later
Reda Chhaibi, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Are GAN Biased? Evaluating GAN-Generated Facial Images via Crowdsourcing
Watch later
Hangzhi Guo, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
a-ReQ: Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
SageMix: Saliency-Guided Mixup for Point Clouds
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%