Differentially Private Episodic Reinforcement Learning with Heavy-tailed Rewards
Jul 24, 2023
Speakers
Xingyu Zhou
Speaker · 0 followers
Sayak Ray Chowdhury
Speaker · 0 followers
About
In this paper we study the problem of (finite horizon tabular) Markov decision processes (MDPs) with heavy-tailed rewards under the constraint of differential privacy (DP). Compared with the previous studies for private reinforcement learning that typically assume rewards are sampled from some bounded or sub-Gaussian distributions to ensure DP, we consider the setting where reward distributions have only finite (1+v)-th moments with some v ∈ (0,1]. By resorting to robust mean estimators for rewards, we first propose two frameworks for heavy-tailed MDPs, i.e., one is for value iteration and another is for policy optimization. Under each framework, we consider both joint differential privacy (JDP) and local differential privacy (LDP) models. Based on our frameworks, we provide regret upper bounds for both JDP and LDP cases, and show that the moment of distributions and privacy budget have significant impact on regrets. Finally, we establish a lower bound of regret minimization for heavy-tailed MDPs in JDP model by reducing it to the instance-independent lower bound of heavy-tailed multi-armed bandits in DP model. We also show the lower bound for the problem in LDP by adopting some private minimax methods. Our results reveal that there are fundamental differences between the problem of private RL with sub-Gaussian and that with heavy-tailed rewards.In this paper we study the problem of (finite horizon tabular) Markov decision processes (MDPs) with heavy-tailed rewards under the constraint of differential privacy (DP). Compared with the previous studies for private reinforcement learning that typically assume rewards are sampled from some bounded or sub-Gaussian distributions to ensure DP, we consider the setting where reward distributions have only finite (1+v)-th moments with some v ∈ (0,1]. By resorting to robust mean estimators for rewa…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Multi-class Graph Clustering via Approximated Effective p-Resistance
Watch later
Shota Saito, …
Generalizing the Gumbel-Softmax with Stochastic Softmax Tricks
Watch later
Structured Cooperative Learning with Graphical Model Priors
Watch later
Concept Learning Across Domains and Modalities
Watch later
Learning Control by Iterative Inversion
Watch later
CLIPood: Generalizing CLIP to Out-of-Distributions
Watch later
Yang Shu, …