A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning
Nov 28, 2022
Speakers
Xidong Feng
Sprecher:in · 0 Follower:innen
Haifeng Zhang
Sprecher:in · 0 Follower:innen
Yaodong Yang
Sprecher:in · 1 Follower:in
About
Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually biased. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper bound of 𝒪(Kα^Kσ̂_In|τ|^-0.5) w.r.t. inner-loop update step K, learning rate α, estimate variance σ̂^2_In and sample size |τ|, and 2) the multi-step Hessian estimation bias Δ̂_H due to the use of autodiff, which has a polynomial impact 𝒪((K-1)(Δ̂_H)^K-1) on the meta-gradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general.Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually biased. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by t…
Organizer
NeurIPS 2022
Konto · 961 Follower:innen
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Provable Benefits of Representational Transfer in RL
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Exploration-Guided Reward Shaping for Reinforcement Learning under Sparse Rewards
Später ansehen
Rati Devidze, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Multi-dataset Training of Transformers for Robust Action Recognition
Später ansehen
Junwei Liang, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
You Never Stop Dancing: Non-freezing Dance Generation via Bank-constrained Manifold Projection
Später ansehen
Jiangxin Sun, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Sparse Probabilistic Circuits via Pruning and Growing
Später ansehen
Meihua Dang, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Conformal Prediction with Temporal Quantile Adjustments
Später ansehen
Zhen Lin, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%