Differentially Private Episodic Reinforcement Learning with Heavy-tailed Rewards
Jul 24, 2023
Speakers
Xingyu Zhou
Speaker · 0 followers
Sayak Ray Chowdhury
Speaker · 0 followers
About
In this paper we study the problem of (finite horizon tabular) Markov decision processes (MDPs) with heavy-tailed rewards under the constraint of differential privacy (DP). Compared with the previous studies for private reinforcement learning that typically assume rewards are sampled from some bounded or sub-Gaussian distributions to ensure DP, we consider the setting where reward distributions have only finite (1+v)-th moments with some v ∈ (0,1]. By resorting to robust mean estimators for rewards, we first propose two frameworks for heavy-tailed MDPs, i.e., one is for value iteration and another is for policy optimization. Under each framework, we consider both joint differential privacy (JDP) and local differential privacy (LDP) models. Based on our frameworks, we provide regret upper bounds for both JDP and LDP cases, and show that the moment of distributions and privacy budget have significant impact on regrets. Finally, we establish a lower bound of regret minimization for heavy-tailed MDPs in JDP model by reducing it to the instance-independent lower bound of heavy-tailed multi-armed bandits in DP model. We also show the lower bound for the problem in LDP by adopting some private minimax methods. Our results reveal that there are fundamental differences between the problem of private RL with sub-Gaussian and that with heavy-tailed rewards.In this paper we study the problem of (finite horizon tabular) Markov decision processes (MDPs) with heavy-tailed rewards under the constraint of differential privacy (DP). Compared with the previous studies for private reinforcement learning that typically assume rewards are sampled from some bounded or sub-Gaussian distributions to ensure DP, we consider the setting where reward distributions have only finite (1+v)-th moments with some v ∈ (0,1]. By resorting to robust mean estimators for rewa…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Leveraging Label Non-Uniformity for Node Classification in Graph Neural Networks
Watch later
Feng Ji, …
Integrating Prior Knowledge in Contrastive Learning
Watch later
Autoregressive Diffusion Model for Graph Generation
Watch later
Lingkai Kong, …
Multi-task Hierarchical Adversarial Inverse Reinforcement Learning
Watch later
Jiayu Chen, …
Transformed Distribution Matching for Missing Value Imputation
Watch later
He Zhao, …
Equivariant Architectures for Learning in Deep Weight Spaces
Watch later
Aviv Navon, …