Efficient Offline Policy Optimization with a Learned Model
Dez 2, 2022
Sprecher:innen
Zichen Liu
Sprecher:in · 0 Follower:innen
Wee Sun Lee
Sprecher:in · 1 Follower:in
Shuicheng Yan
Sprecher:in · 0 Follower:innen
Zhongwen Xu
Sprecher:in · 0 Follower:innen
Über
MuZero Unplugged presents a promising approach for offline policy learning from logged data. It conducts Monte-Carlo Tree Search (MCTS) with a learned model and leverages Reanalyze algorithm to learn purely from offline data. For good performance, MCTS requires accurate learned models and a large number of simulations, thus costing huge computing time. This paper investigates a few hypotheses where MuZero Unplugged may not work well under the offline RL settings, including 1) learning with limited data coverage; 2) learning from offline data of stochastic environments; 3) improperly parameterized models given the offline data; 4) with a low compute budget. We propose to use a regularized one-step look-ahead approach to tackle the above issues. Instead of planning with the expensive MCTS, we use the learned model to construct an advantage estimation based on a one-step rollout. Policy improvements are towards the direction that maximizes the estimated advantage with regularization of the dataset. We conduct extensive empirical studies with BSuite environments to verify the hypotheses and then run our algorithm on the RL Unplugged Atari benchmark. Experimental results show that our proposed approach achieves stable performance even with an inaccurate learned model. On the large-scale Atari benchmark, the proposed method outperforms MuZero Unplugged by 43MuZero Unplugged presents a promising approach for offline policy learning from logged data. It conducts Monte-Carlo Tree Search (MCTS) with a learned model and leverages Reanalyze algorithm to learn purely from offline data. For good performance, MCTS requires accurate learned models and a large number of simulations, thus costing huge computing time. This paper investigates a few hypotheses where MuZero Unplugged may not work well under the offline RL settings, including 1) learning with limit…
Organisator
NeurIPS 2022
Konto · 961 Follower:innen
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
Subsidiary Prototype Alignment for Universal Domain Adaptation
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
ForestBench: Equitable Benchmarks for Monitoring, Reporting, and Verification of Nature-Based Solutions with Machine Learning
Später ansehen
Lucas Czech, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Accelerating Perturbed Stochastic Iterates in Asynchronous Lock-Free Optimization
Später ansehen
Kaiwen Zhou, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Q A
Später ansehen
Sayak Paul, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Peripheral Vision Transformer
Später ansehen
Juhong Min, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Rethinking Alignment in Video Super-Resolution Transformers
Später ansehen
Shuwei Shi, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%