A Connection between One-Step RL and Critic Regularization in Reinforcement Learning
Jul 24, 2023
Speakers
Benjamin Eysenbach
Speaker · 0 followers
Matthieu Geist
Speaker · 0 followers
Sergey Levine
Speaker · 1 follower
Russ Salakhutdinov
Speaker · 13 followers
About
As with any machine learning problem with limited data, effective offline RL algorithms require careful regularization to avoid overfitting. One class of methods, known as one-step RL, perform just one step of policy improvement. These methods, which include advantage-weighted regression and conditional behavioral cloning, are thus simple and stable, but can have limited asymptotic performance. A second class of methods, known as critic regularization, perform many steps of policy improvement with a regularized objective. These methods typically require more compute but have appealing lower-bound guarantees. In this paper, we draw a connection between these methods: applying a multi-step critic regularization method with a regularization coefficient of 1 yields the same policy as one-step RL. While our theoretical results require assumptions (e.g., deterministic dynamics), our experiments nevertheless show that our analysis makes accurate, testable predictions about practical offline RL methods (CQL and one-step RL) with commonly-used hyperparameters.As with any machine learning problem with limited data, effective offline RL algorithms require careful regularization to avoid overfitting. One class of methods, known as one-step RL, perform just one step of policy improvement. These methods, which include advantage-weighted regression and conditional behavioral cloning, are thus simple and stable, but can have limited asymptotic performance. A second class of methods, known as critic regularization, perform many steps of policy improvement wi…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
A Kernel-Based View of Language Model Fine-Tuning
Watch later
Sample Complexity Bounds for Learning High-dimensional Simplices in Noisy Regimes
Watch later
DIVISION: Memory Efficient Training via Dual Activation Precision
Watch later
Guanchu Wang, …
Generalization and Corruption Resistance via Distributionally Robust Optimization
Watch later
