Robust Situational Reinforcement Learning in Face of Context Disturbances
Jul 24, 2023
Speakers
Jinpeng Zhang
Speaker · 0 followers
Yufeng Zheng
Speaker · 0 followers
Chuheng Zhang
Speaker · 0 followers
Jiang Bian
Speaker · 0 followers
About
In many real-world tasks, some parts of state features, called contexts, are independent of action signals, e.g., customer demand in inventory control, speed of lead car in autonomous driving, etc. One of the challenges of reinforcement learning in these applications is that the true context transitions can be easily exposed some unknown source of contamination, leading to a shift of context transitions between source domains and target domains, which could cause performance degradation for RL algorithms. However, existing methods on robust RL aim at learning robust policies against the deviations of the entire system dynamics. To tackle this problem, this paper proposes the framework of robust situational Markov decision process (RS-MDP) which captures the possible deviations of context transitions explicitly. To scale to large context space, we introduce the softmin smoothed robust Bellman operator to learn the robust Q-value approximately, and apply our RS-MDP framework to existing RL algorithm SAC to learn the desired robust policies. We conduct experiments on several robot control tasks with dynamic contexts and inventory control tasks to demonstrate that our algorithm can generalize better and more robust against deviations of context transitions, and outperform existing robust RL algorithms.In many real-world tasks, some parts of state features, called contexts, are independent of action signals, e.g., customer demand in inventory control, speed of lead car in autonomous driving, etc. One of the challenges of reinforcement learning in these applications is that the true context transitions can be easily exposed some unknown source of contamination, leading to a shift of context transitions between source domains and target domains, which could cause performance degradation for RL a…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Leveraging Large Scale Models for Identifying and Fixing Deep Neural Networks Biases
Watch later
Discover and Cure: Concept-aware Mitigation of Spurious Correlation
Watch later
Shirley Wu, …
Learning Rate Schedules in the Presence of Distribution Shift
Watch later
Policy Gradient in Robust MDPs with Global Convergence Guarantee
Watch later
Qiuhao Wang, …