The Emphatic Approach to Average-Reward Policy Evaluation
2. prosince 2022
Řečníci
O prezentaci
Off-policy policy evaluation has been a longstanding problem in reinforcement learning. This paper looks at this problem under the average-reward formulation with function approximation. Differential temporal-difference (TD) learning has been proposed recently and has shown great potential compared to previous average-reward learning algorithms. In the tabular setting, off-policy differential TD is guaranteed to converge. However, the convergence guarantee cannot be carried through the function approximation setting. To address the instability of off-policy differential TD, we investigate the emphatic approach proposed for the discounted formulation. Specifically, we introduce average emphatic trace for average-reward off-policy learning. We further show that without any variance reduction techniques, the new trace suffers from slow learning due to high variance of importance sampling ratios. Finally, we show that differential emphatic TD(β), extended from the discounted setting, can save us from the high variance while introducing bias. Experimental results on a counterexample show that differential emphatic TD(β) performs better than an existing competitive off-policy algorithm.Off-policy policy evaluation has been a longstanding problem in reinforcement learning. This paper looks at this problem under the average-reward formulation with function approximation. Differential temporal-difference (TD) learning has been proposed recently and has shown great potential compared to previous average-reward learning algorithms. In the tabular setting, off-policy differential TD is guaranteed to converge. However, the convergence guarantee cannot be carried through the function…
Organizátor
NeurIPS 2022
Konto · 961 Follower:innen
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Discovered Policy Optimisation
Später ansehen
Chris Lu, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Vision GNN: An Image is Worth Graph of Nodes
Später ansehen
Kai Han, …
Ewigspeicher-Fortschrittswert: 1 = 0.1%
One Layer is All You Need
Später ansehen
Yue Bai, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing
Später ansehen
Yonggan Fu, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Combinatorial Bandits with Linear Constraints: Beyond Knapsacks and Fairness
Später ansehen
Qingsong Liu, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Accelerating Perturbed Stochastic Iterates in Asynchronous Lock-Free Optimization
Später ansehen
Kaiwen Zhou, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%