When to Update Your Model: Constrained Model-based Reinforcement Learning
Nov 28, 2022
Speakers
Tianying Ji
Speaker · 0 followers
Fuchun Sun
Speaker · 0 followers
Mingxuan Jing
Speaker · 0 followers
Fengxiang He
Speaker · 0 followers
Wenbing Huang
Speaker · 0 followers
About
Designing and analyzing model-based RL (MBRL) algorithms with guaranteed monotonic improvement has been challenging, mainly due to the interdependence between policy optimization and model learning. Existing discrepancy bounds generally ignore the impacts of model shifts, and their corresponding algorithms are prone to degrade performance by drastic model updating. In this work, we first propose a novel and general theoretical scheme for a non-decreasing performance guarantee of MBRL. Our follow-up derived bounds reveal the relationship between model shifts and performance improvement. These discoveries encourage us to formulate a constrained lower-bound optimization problem to permit the monotonicity of MBRL. A further example demonstrates that learning models from a dynamically-varying number of explorations benefit the eventual returns. Motivated by these analyses, we design a simple but effective algorithm CMLO (Constrained Model-shift Lower-bound Optimization), by introducing an event-triggered mechanism that flexibly determines when to update the model. Experiments show that CMLO surpasses other state-of-the-art methods and produces a boost when various policy optimization methods are employed.Designing and analyzing model-based RL (MBRL) algorithms with guaranteed monotonic improvement has been challenging, mainly due to the interdependence between policy optimization and model learning. Existing discrepancy bounds generally ignore the impacts of model shifts, and their corresponding algorithms are prone to degrade performance by drastic model updating. In this work, we first propose a novel and general theoretical scheme for a non-decreasing performance guarantee of MBRL. Our follow…
Organizer
NeurIPS 2022
Account · 962 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Performance and utility trade-off in interpretable sleep staging
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The effects of gender bias in word embeddings on depression prediction
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Coresets for Relational Data and The Applications
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Panel RL Implementation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Building a Subspace of Policies for Scalable Continual Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Geometric Order Learning for Rank Estimation
Watch later
Seon-Ho Lee, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%