A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning
Nov 28, 2022
Sprecher:innen
Über
Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually biased. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper bound of 𝒪(Kα^Kσ̂_In|τ|^-0.5) w.r.t. inner-loop update step K, learning rate α, estimate variance σ̂^2_In and sample size |τ|, and 2) the multi-step Hessian estimation bias Δ̂_H due to the use of autodiff, which has a polynomial impact 𝒪((K-1)(Δ̂_H)^K-1) on the meta-gradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general.Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually biased. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by t…
Organisator
NeurIPS 2022
Účet · 961 sledujících
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
A Finite-Particle Convergence Rate for Stein Variational Gradient Descent
Zhlédnout později
Jiaxin Shi, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Score-based Generative Models and Their Applications
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Making Sense of Dependence: Efficient Black-box Explanations Using Dependence Measure
Zhlédnout později
Paul Novello, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Near-Optimal Randomized Exploration for Tabular Markov Decision Processes
Zhlédnout později
Zhihan Xiong, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Causal Analysis of the TOPCAT Trial: Spironolactone for Preserved Cardiac Function Heart Failure
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 1 diváků, což je 0.1 %
Biologically plausible solutions for spiking networks with efficient coding
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %