Contrastive Value Learning: Implicit Models for Simple Offline RL
Dec 2, 2022
Speakers
Bogdan Mazoure
Speaker · 0 followers
Benjamin Eysenbach
Speaker · 0 followers
Ofir Nachum
Speaker · 2 followers
Jonathan Tompson
Speaker · 0 followers
About
Model-based reinforcement learning (RL) methods are appealing in the offline setting because they allow an agent to reason about the consequences of actions without interacting with the environment. Prior methods learn a 1-step dynamics model, which predicts the next state given the current state and action. These models do not immediately tell the agent which actions to take, but must be integrated into a larger RL framework. Can we model the environment dynamics in a different way, such that the learned model does directly indicate the value of each action? In this paper, we propose Contrastive Value Learning (CVL), which learns an implicit, multi-step model of the environment dynamics. This model can be learned without access to reward functions, but nonetheless can be used to directly estimate the value of each action, without requiring any TD learning. Because this model represents the multi-step transitions implicitly, it avoids having to predict high-dimensional observations and thus scales to high-dimensional tasks. Our experiments demonstrate that CVL outperforms prior offline RL methods on complex continuous control benchmarks.Model-based reinforcement learning (RL) methods are appealing in the offline setting because they allow an agent to reason about the consequences of actions without interacting with the environment. Prior methods learn a 1-step dynamics model, which predicts the next state given the current state and action. These models do not immediately tell the agent which actions to take, but must be integrated into a larger RL framework. Can we model the environment dynamics in a different way, such that t…
Organizer
NeurIPS 2022
Account · 961 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
On the Symmetries of Deep Learning Models and their Internal Representations
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
When are Local Queries Useful for Robust Learning?
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Exploitability Minimization in Games and Beyond
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Generalization Analysis of Message Passing Neural Networks on Large Random Graphs
Watch later
Sohir Maskey, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Predictive Coding beyond Gaussian Distributions
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Efficient and Effective Multi-task Grouping via Meta Learning on Task Combinations
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%