A Near-Optimal Algorithm for Safe Reinforcement Learning Under Instantaneous Hard Constraints
Jul 24, 2023
Sprecher:innen
Yingbin Liang
Sprecher:in · 0 Follower:innen
Ness Shroff
Sprecher:in · 0 Follower:innen
Über
In many applications of Reinforcement Learning (RL), it is critically important that the algorithm performs safely, such that instantaneous hard constraints are satisfied at each step, and unsafe states and actions are avoided. However, existing algorithms for "safe" RL are often designed under constraints that either require expected cumulative costs to be bounded or assume all states are safe. Thus, such algorithms could violate instantaneous hard constraints and traverse unsafe states (and actions) in practice. Therefore, in this paper, we develop the first near-optimal safe RL algorithm for episodic Markov Decision Processes with unsafe states and actions under instantaneous hard constraints and the linear mixture model. It not only achieves a regret Õ(dH^3 √(dK)/Δ_c) that tightly matches the state-of-the-art regret in the setting with only unsafe actions and nearly matches that in the unconstrained setting, but is also safe at each step, where d is the feature-mapping dimension, K is the number of episodes, H is the number of steps in each episode, and Δ_c is a safety-related parameter. We also provide a lower bound Ω̃(max dH √(K), H/Δ_c^2 ), which indicates that the dependency on Δ_c is necessary. Further, both our algorithm design and regret analysis involve several novel ideas, which may be of independent interest.In many applications of Reinforcement Learning (RL), it is critically important that the algorithm performs safely, such that instantaneous hard constraints are satisfied at each step, and unsafe states and actions are avoided. However, existing algorithms for "safe" RL are often designed under constraints that either require expected cumulative costs to be bounded or assume all states are safe. Thus, such algorithms could violate instantaneous hard constraints and traverse unsafe states (and ac…
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
Repository-Level Prompt Generation for Large Language Models of Code
Später ansehen
Calibrating Multimodal Learning
Später ansehen
Huan Ma, …
On the Statistical Benefits of Temporal Difference Learning
Später ansehen
Improving Hyperparameter Learning under Approximate Inference in Gaussian Process Models
Später ansehen
Rui Li, …
Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks
Später ansehen
