The Benefits of Model-Based Generalization in Reinforcement Learning
24. července 2023
Řečníci
Kenny Young
Sprecher:in · 0 Follower:innen
Aditya Ramesh
Sprecher:in · 1 Follower:in
Louis Kirsch
Sprecher:in · 0 Follower:innen
Jürgen Schmidhuber
Sprecher:in · 2 Follower:innen
O prezentaci
Model-Based Reinforcement Learning (RL) is widely believed to have the potential to improve sample efficiency by allowing an agent to synthesize large amounts of imagined experience. Experience Replay (ER) can be considered a simple kind of model, which has proved effective at improving the stability and efficiency of deep RL. In principle, a learned parametric model could improve on ER by generalizing from real experience to augment the dataset with additional plausible experience. However, given that learned value functions can also generalize, it is not immediately obvious why model generalization should be better. Here, we provide theoretical and empirical insight into when, and how, we can expect data generated by a learned model to be useful. First, we provide a simple theorem motivating how learning a model as an intermediate step can narrow down the set of possible value functions more than learning a value function directly from data using the Bellman equation. Second, we provide an illustrative example showing empirically how a similar effect occurs in a more concrete setting with neural network function approximation. Finally, we provide extensive experiments showing the benefit of model-based learning for online RL in environments with combinatorial complexity, but factored structure that allows a learned model to generalize. In these experiments, we take care to control for other factors in order to isolate, insofar as possible, the benefit of using experience generated by a learned model relative to ER alone.Model-Based Reinforcement Learning (RL) is widely believed to have the potential to improve sample efficiency by allowing an agent to synthesize large amounts of imagined experience. Experience Replay (ER) can be considered a simple kind of model, which has proved effective at improving the stability and efficiency of deep RL. In principle, a learned parametric model could improve on ER by generalizing from real experience to augment the dataset with additional plausible experience. However, giv…
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Data-Efficient Contrastive Self-supervised Learning
Später ansehen
Reliable Measures of Spread in High Dimensional Latent Spaces
Später ansehen
Anna Marbut, …
Improving Open Language Models by Learning from Organic Interactions
Später ansehen
Jason Weston, …
Open-Vocabulary Universal Image Segmentation with MaskCLIP
Später ansehen
Zheng Ding, …