Responsive Safety in RL by PID Lagrangian Methods

Jul 12, 2020



CMDPs formalize the problem of safe reinforcement learning by exposing a cost signal alongside the reward and limiting its accumulation. Lagrangian method are the most commonly used algorithms for the resulting constrained optimization problem. Yet they are known to oscillate and overshoot cost limits, causing constraint-violating behavior during training. In this paper, we aim to correct this shortcoming. We begin by proposing a novel modification to the classic Lagrangian method: we add a “proportional” term to the Lagrange multiplier update and show that it induces favorable learning dynamics through damping. This intuition leads to our introduction of PID control for the Lagrange multiplier in constrained RL, which we cast as a dynamical system. We conduct extensive experiments in a deep RL setting, in which our methods set a new state of the art by dramatically reducing constraint violations while maintaining high returns. Moreover, we show significant improvements in robustness to hyperparameters. Unlike other recent algorithms, ours remains nearly as simple to derive and implement as the baseline Lagrangian method.



About ICML 2020

The International Conference on Machine Learning (ICML) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence known as machine learning. ICML is globally renowned for presenting and publishing cutting-edge research on all aspects of machine learning used in closely related areas like artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, and robotics. ICML is one of the fastest growing artificial intelligence conferences in the world. Participants at ICML span a wide range of backgrounds, from academic and industrial researchers, to entrepreneurs and engineers, to graduate students and postdocs.

Store presentation

Should this presentation be stored for 1000 years?

How do we store presentations

Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%


Recommended Videos

Presentations on similar topic, category or speaker