The Benefits of Model-Based Generalization in Reinforcement Learning
2. prosince 2022
Řečníci
Kenny Young
Řečník · 0 sledujících
Aditya Ramesh
Řečník · 1 sledující
Louis Kirsch
Řečník · 0 sledujících
Jürgen Schmidhuber
Řečník · 2 sledující
O prezentaci
Model-Based Reinforcement Learning (RL) is widely believed to have the potential to improve sample efficiency by allowing an agent to synthesize large amounts of imagined experience. Experience Replay (ER) can be considered a simple kind of model, which has proved extremely effective at improving the stability and efficiency of deep RL. In principle, a learned parametric model could improve on ER by generalizing from real experience to augment the dataset with additional plausible experience. However, owing to the many design choices involved in empirically successful algorithms, it can be very hard to establish where the benefits are actually coming from. Here, we provide theoretical and empirical insight into when, and how, we can expect data generated by a learned model to be useful. First, we provide a general theorem motivating how learning a model as an intermediate step can narrow down the set of possible value functions more than learning a value function directly from data using the Bellman equation. Second, we provide an illustrative example showing empirically how a similar effect occurs in a more concrete setting with neural network function approximation. Finally, we provide extensive experiments showing the benefit of model-based learning for online RL in environments with combinatorial complexity, but factored structure that allows a learned model to generalize. In these experiments, we take care to control for other factors in order to isolate, insofar as possible, the benefit of using experience generated by a learned model relative to ER alone.Model-Based Reinforcement Learning (RL) is widely believed to have the potential to improve sample efficiency by allowing an agent to synthesize large amounts of imagined experience. Experience Replay (ER) can be considered a simple kind of model, which has proved extremely effective at improving the stability and efficiency of deep RL. In principle, a learned parametric model could improve on ER by generalizing from real experience to augment the dataset with additional plausible experience. Ho…
Organizátor
NeurIPS 2022
Účet · 962 sledujících
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Democratizing surcigl skills - video analysis for quantifying surgical expertise
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics
Zhlédnout později
Lianhui Qin, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Machine Learning for Climate Risk
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Accelerated Sampling and Improved Synthesis in Diffusion Models
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Target alignment in truncated kernel ridge regression
Zhlédnout později
Arash Amini, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Learning Counterfactually Invariant Predictors
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %