Policy Aware Model Learning via Transition Occupancy Matching
Dec 2, 2022
Speakers
Jason Yecheng Ma
Speaker · 0 followers
Kausik Sivakumar
Speaker · 0 followers
Osbert Bastani
Speaker · 0 followers
Dinesh Jayaraman
Speaker · 0 followers
About
Model-based reinforcement learning (MBRL) is an effective paradigm for sample-efficient policy learning. The pre-dominant MBRL strategy iteratively learns the dynamics model by performing maximum likelihood (MLE) on the entire replay buffer and trains the policy using fictitious transitions from the learned model. Given that not all transitions in the replay buffer are equally informative about the task or the policy's current progress, this MLE strategy cannot be optimal and bears no clear relation to the standard RL objective. In this work, we propose Transition Occupancy Matching (TOM), a policy-aware model learning algorithm that maximizes a lower bound on the standard RL objective. TOM learns a policy-aware dynamics model by minimizing an f-divergence between the distribution of transitions that the current policy visits in the real environment and in the learned model; then, the policy can be updated using any pre-existing RL algorithm with log-transformed reward. TOM's practical implementation builds on tools from dual reinforcement learning and learns the optimal transition occupancy ratio between the current policy and the replay buffer; leveraging this ratio as importance weights, TOM amounts to performing MLE model learning on the correct, policy aware transition distribution. Crucially, TOM is a model learning sub-routine and is compatible with any backbone MBRL algorithm that implements MLE-based model learning. On the standard set of Mujoco locomotion tasks, we find TOM improves the learning speed of a standard MBRL algorithm and can reach the same asymptotic performance with as much as 50Model-based reinforcement learning (MBRL) is an effective paradigm for sample-efficient policy learning. The pre-dominant MBRL strategy iteratively learns the dynamics model by performing maximum likelihood (MLE) on the entire replay buffer and trains the policy using fictitious transitions from the learned model. Given that not all transitions in the replay buffer are equally informative about the task or the policy's current progress, this MLE strategy cannot be optimal and bears no clear rela…
Organizer
NeurIPS 2022
Account · 961 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Sequential Information Design: Learning to Persuade in the Dark
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Deep Combinatorial Aggregation
Watch later
Yuesong Shen, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Machine Learning for Predicting Climate Extremes
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Teaching Algorithmic Reasoning via In-context Learning
Watch later
Hattie Zhou, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Alternating Mirror Descent for Constrained Min-Max Games
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Bridging the Gap: Unifying the Training and Evaluation of Neural Network Binary Classifiers
Watch later
Nathan Tsoi, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%