A Study of Global and Episodic Bonuses for Exploration in Contextual MDPs
Jul 25, 2023
Speakers
Mikael Henaff
Speaker · 0 followers
Minqi Jiang
Speaker · 0 followers
Roberta Raileanu
Speaker · 0 followers
About
Exploration in environments which differ across episodes has received increasing attention in recent years. Current methods use some combination of global novelty bonuses, computed using the agent's entire training experience, and episodic novelty bonuses, computed using only experience from the current episode. However, the use of these two types of bonuses has been ad-hoc and poorly understood. In this work, we shed light on the behavior of these two types of bonuses through controlled experiments on easily interpretable tasks as well as challenging pixel-based settings. We find that the two types of bonuses succeed in different settings, with episodic bonuses being most effective when there is little shared structure across episodes and global bonuses being effective when more structure is shared. We develop a conceptual framework which makes this notion of shared structure precise by considering the variance of the value function across contexts, and which provides a unifying explanation of our empirical results. We furthermore find that combining the two bonuses can lead to more robust performance across different degrees of shared structure, and investigate different algorithmic choices for defining and combining global and episodic bonuses based on function approximation. This results in an algorithm which sets a new state of the art across 16 tasks from the MiniHack suite used in prior work, and also performs robustly on Habitat and Montezuma's Revenge.Exploration in environments which differ across episodes has received increasing attention in recent years. Current methods use some combination of global novelty bonuses, computed using the agent's entire training experience, and episodic novelty bonuses, computed using only experience from the current episode. However, the use of these two types of bonuses has been ad-hoc and poorly understood. In this work, we shed light on the behavior of these two types of bonuses through controlled experim…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
MABe22: A Multi-Species Multi-Task Benchmark for Learned Representations of Behavior
Watch later
RényiTester, a Variational Approach to Testing Differential Privacy
Watch later
Causal Bounds in Quasi-Markovian Graphs
Watch later
DIVISION: Memory Efficient Training via Dual Activation Precision
Watch later
Guanchu Wang, …
Automatic Intrinsic Reward Shaping for Exploration in Deep Reinforcement Learning
Watch later
Mingqi Yuan, …
