“Deep Bandits Show-Off”: Simple and Efficient Exploration with Deep Networks
Dez 6, 2021
Sprecher:innen
Mattia Rigotti
Sprecher:in · 0 Follower:innen
Über
Designing efficient exploration is central to Reinforcement Learning due to the fundamental problem posed by the exploration-exploitation dilemma. Bayesian exploration strategies like Thompson Sampling resolve this trade-off in a principled way by modeling and updating the distribution of the parameters of the the action-value function, the outcome model of the environment.However, this technique becomes infeasible for complex environments due to the computational intractability of maintaining probability distributions over parameters of outcome models of corresponding complexity.Moreover, the approximation techniques introduced to mitigate this issue typically result in poor exploration-exploitation trade-offs, as observed in the case of deep neural network models with approximate posterior methods that have been shown to underperform in the deep bandit scenario.In this paper we introduce Sample Average Uncertainty (SAU), a simple and efficient uncertainty measure for contextual bandits.While Bayesian approaches like Thompson Sampling estimate outcomes uncertainty indirectly by first quantifying the variability over the parameters of the outcome model, SAU is a frequentist approach that directly estimates the uncertainty of the outcomes based on the value predictions.Importantly, we show theoretically that the uncertainty measure estimated by SAU asymptotically matches the uncertainty provided by Thompson Sampling, as well as its regret bounds.Because of its simplicity SAU can be seamlessly applied to deep contextual bandits as a very scalable drop-in replacement for epsilon-greedy exploration.We confirm empirically our theory by showing that SAU-based exploration outperforms current state-of-the-art deep Bayesian bandit methods on several real-world datasets at modest computation cost, and make the code to reproduce our results available.Designing efficient exploration is central to Reinforcement Learning due to the fundamental problem posed by the exploration-exploitation dilemma. Bayesian exploration strategies like Thompson Sampling resolve this trade-off in a principled way by modeling and updating the distribution of the parameters of the the action-value function, the outcome model of the environment.However, this technique becomes infeasible for complex environments due to the computational intractability of maintaining p…
Organisator
NeurIPS 2021
Konto · 1,9k Follower:innen
Über NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation
Später ansehen
Lei Ke, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Object Representations Guided By Optical Flow
Später ansehen
Jianing Qian, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Learning Models for Actionable Recourse
Später ansehen
Alexis Ross, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Neural Circuit Synthesis from Specification Patterns
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations
Später ansehen
Jiayao Zhang, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Unbalanced Optimal Transport through Non-negative Penalized Linear Regression
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%