The Benefits of Model-Based Generalization in Reinforcement Learning
Dez 2, 2022
Sprecher:innen
Kenny Young
Řečník · 0 sledujících
Aditya Ramesh
Řečník · 1 sledující
Louis Kirsch
Řečník · 0 sledujících
Jürgen Schmidhuber
Řečník · 2 sledující
Über
Model-Based Reinforcement Learning (RL) is widely believed to have the potential to improve sample efficiency by allowing an agent to synthesize large amounts of imagined experience. Experience Replay (ER) can be considered a simple kind of model, which has proved extremely effective at improving the stability and efficiency of deep RL. In principle, a learned parametric model could improve on ER by generalizing from real experience to augment the dataset with additional plausible experience. However, owing to the many design choices involved in empirically successful algorithms, it can be very hard to establish where the benefits are actually coming from. Here, we provide theoretical and empirical insight into when, and how, we can expect data generated by a learned model to be useful. First, we provide a general theorem motivating how learning a model as an intermediate step can narrow down the set of possible value functions more than learning a value function directly from data using the Bellman equation. Second, we provide an illustrative example showing empirically how a similar effect occurs in a more concrete setting with neural network function approximation. Finally, we provide extensive experiments showing the benefit of model-based learning for online RL in environments with combinatorial complexity, but factored structure that allows a learned model to generalize. In these experiments, we take care to control for other factors in order to isolate, insofar as possible, the benefit of using experience generated by a learned model relative to ER alone.Model-Based Reinforcement Learning (RL) is widely believed to have the potential to improve sample efficiency by allowing an agent to synthesize large amounts of imagined experience. Experience Replay (ER) can be considered a simple kind of model, which has proved extremely effective at improving the stability and efficiency of deep RL. In principle, a learned parametric model could improve on ER by generalizing from real experience to augment the dataset with additional plausible experience. Ho…
Organisator
NeurIPS 2022
Účet · 962 sledujících
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
Sparse Gaussian Process Hyperparameters: Optimize or Integrate?
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
“Today we talk to the machine” - Unveiling data for providing micro-credit loans using conversational systems.
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Panel 3: Geographic disparities in where research is produced
Zhlédnout později
Pablo Castro, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
A Variational Edge Partition Model for Supervised Graph Representation Learning
Zhlédnout později
Yilin He, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Reinforcement Learning with Automated Auxiliary Loss Search
Zhlédnout později
Tairan He, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
LBD: Decouple Relevance and Observation for Individual-Level Unbiased Learning to Rank
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %