Replay Buffer With Local Forgetting for Adaptive Deep Model-Based Reinforcement Learning
Dec 2, 2022
Speakers
Ali Rahimi-Kalahroudi
Speaker · 0 followers
Janarthanan Rajendran
Speaker · 0 followers
Ida Momennejad
Speaker · 0 followers
Harm van Seijen
Speaker · 0 followers
Sarath Chandar
Speaker · 1 follower
About
One of the key behavioral characteristics used in neuroscience to determine whether the subject of study—be it a rodent or a human—exhibits model-based learning is effective adaptation to local changes in the environment. In reinforcement learning, however, recent work has shown that modern deep model-based reinforcement-learning (MBRL) methods adapt poorly to such changes. An explanation for this mismatch is that MBRL methods are typically designed with sample-efficiency on a single task in mind and the requirements for effective adaptation are substantially higher, both in terms of the learned world model and the planning routine. One particularly challenging requirement is that the learned world model has to be sufficiently accurate throughout relevant parts of the state-space. This is challenging for deep-learning-based world models due to catastrophic forgetting. And while a replay buffer can mitigate the effects of catastrophic forgetting, the traditional first-in-first-out replay buffer precludes effective adaptation due to maintaining stale data. In this work, we show that a conceptually simple variation of this traditional replay buffer is able to overcome this limitation. By removing only samples from the buffer from the local neighbourhood of the newly observed samples, deep world models can be built that maintain their accuracy across the state-space, while also being able to effectively adapt to changes in the reward function. We demonstrate this by applying our replay-buffer variation to the classical Dyna method, as well as to recent methods such as PlaNet and DreamerV2, showing for the first time that deep model-based methods are able to achieve effective adaptation.One of the key behavioral characteristics used in neuroscience to determine whether the subject of study—be it a rodent or a human—exhibits model-based learning is effective adaptation to local changes in the environment. In reinforcement learning, however, recent work has shown that modern deep model-based reinforcement-learning (MBRL) methods adapt poorly to such changes. An explanation for this mismatch is that MBRL methods are typically designed with sample-efficiency on a single task in min…
Organizer
NeurIPS 2022
Account · 961 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Dimenison-Reduced Adaptive Gradient Method
Watch later
Jingyang Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Mildly Conservative Q-Learning for Offline Reinforcement Learning
Watch later
Jiafei Lyu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
General Cutting Planes for Bound-Propagation-Based Neural Network Verification
Watch later
Huan Zhang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Momentum Boosted Episodic Memory for Improving Learning in Long-Tailed RL Environments
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Feasible Adversarial Robust Reinforcement Learning
Watch later
JB Lanier, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Private Synthetic Data for Multitask Learning and Marginal Queries
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%