The Benefits of Model-Based Generalization in Reinforcement Learning
Dec 2, 2022
Speakers
Kenny Young
Speaker · 0 followers
Aditya Ramesh
Speaker · 1 follower
Louis Kirsch
Speaker · 0 followers
Jürgen Schmidhuber
Speaker · 2 followers
About
Model-Based Reinforcement Learning (RL) is widely believed to have the potential to improve sample efficiency by allowing an agent to synthesize large amounts of imagined experience. Experience Replay (ER) can be considered a simple kind of model, which has proved extremely effective at improving the stability and efficiency of deep RL. In principle, a learned parametric model could improve on ER by generalizing from real experience to augment the dataset with additional plausible experience. However, owing to the many design choices involved in empirically successful algorithms, it can be very hard to establish where the benefits are actually coming from. Here, we provide theoretical and empirical insight into when, and how, we can expect data generated by a learned model to be useful. First, we provide a general theorem motivating how learning a model as an intermediate step can narrow down the set of possible value functions more than learning a value function directly from data using the Bellman equation. Second, we provide an illustrative example showing empirically how a similar effect occurs in a more concrete setting with neural network function approximation. Finally, we provide extensive experiments showing the benefit of model-based learning for online RL in environments with combinatorial complexity, but factored structure that allows a learned model to generalize. In these experiments, we take care to control for other factors in order to isolate, insofar as possible, the benefit of using experience generated by a learned model relative to ER alone.Model-Based Reinforcement Learning (RL) is widely believed to have the potential to improve sample efficiency by allowing an agent to synthesize large amounts of imagined experience. Experience Replay (ER) can be considered a simple kind of model, which has proved extremely effective at improving the stability and efficiency of deep RL. In principle, a learned parametric model could improve on ER by generalizing from real experience to augment the dataset with additional plausible experience. Ho…
Organizer
NeurIPS 2022
Account · 961 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
SKFlow: Learning Optical Flow with Super Kernels
Watch later
Shangkun Sun, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Improving Out-of-distribution Generalization by Adversarial Training with Structured Priors
Watch later
Qixun Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
AI and Quantum Computing
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Decentralized Mechanism Design
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Stronger is not better: Better Augmentations in Contrastive Learning for Medical Image Segmentation
Watch later
Azeez Idris, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Efficient Non-Parametric Optimizer Search for Diverse Tasks
Watch later
Ruochen Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%