Variance Reduction in Off-Policy Deep Reinforcement Learning using Spectral Normalization
Dec 2, 2022
Speakers
Payal Bawa
Speaker · 0 followers
Rafael Oliveira
Speaker · 0 followers
Fabio Ramos
Speaker · 0 followers
About
Off-policy deep reinforcement learning algorithms like Soft Actor Critic (SAC) have achieved state-of-the-art results in several high dimensional continuous control tasks. Despite their success, they are prone to instability due to the deadly triad of off-policy training, function approximation, and bootstrapping. Unstable training of off-policy algorithms leads to sample inefficient and sub-optimal asymptotic performance, thus preventing their real-world deployment. To mitigate these issues, previously proposed solutions have focused on advances like target networks to alleviate instability and the introduction of twin critics to address overestimation bias. However, these modifications fail to address the issue of noisy gradient estimation with excessive variance, resulting in instability and slow convergence. Our proposed method, Spectral Normalized Actor Critic (SNAC), regularizes the actor and the critics using spectral normalization to systematically bound the gradient norm. Spectral normalization constrains the magnitudes of the gradients resulting in smoother actor-critics with robust and sample-efficient performance thus making them suitable for deployment in stability-critical and compute-constrained applications. We present empirical results on several challenging reinforcement learning benchmarks and extensive ablation studies to demonstrate the effectiveness of our proposed method.Off-policy deep reinforcement learning algorithms like Soft Actor Critic (SAC) have achieved state-of-the-art results in several high dimensional continuous control tasks. Despite their success, they are prone to instability due to the deadly triad of off-policy training, function approximation, and bootstrapping. Unstable training of off-policy algorithms leads to sample inefficient and sub-optimal asymptotic performance, thus preventing their real-world deployment. To mitigate these issues, pr…
Organizer
NeurIPS 2022
Account · 961 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer
Watch later
Sen Lin, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Bias Amplification in Image Classification
Watch later
Melissa Hall, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Self-supervised Heterogeneous Graph Pre-training Based on Structural Clustering
Watch later
Yaming Yang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metrics
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The Phenomenon of Policy Churn
Watch later
Tom Schaul, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Effectiveness of Vision Transformer for Fast and Accurate Single-Stage Pedestrian Detector
Watch later
Jing Yuan, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%