Agents must be able to adapt quickly as an environment changes. We find that existing model-based reinforcement learning agents are unable to do this well, in part because of the way that they leverage their experience to train a world model. In particular, strategies that are effective for updating world models in unchanging environments, such as uniform sampling from a replay buffer, can become ineffective in changing environments. Here, we present Curious Replay, a method for improving the ab…