Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings
Dec 6, 2021
Speakers
Yu-Xiang Wang
Speaker · 0 followers
About
This work studies the statistical limits of uniform convergence for offline policy evaluation (OPE) problems with model-based methods (for episodic MDP) and provides a unified framework towards optimal learning for several well-motivated offline tasks. Uniform OPE sup_Π|Q^π-Q̂^π|<ϵ is a stronger measure than the point-wise OPE and ensures offline learning when Π contains all policies (the global class). In this paper, we establish an Ω(H^2 S/d_mϵ^2) lower bound (over model-based family) for the global uniform OPE and our main result establishes an upper bound of Õ(H^2/d_mϵ^2) for the local uniform convergence that applies to all near-empirically optimal policies for the MDPs with stationary transition. Here d_m is the minimal marginal state-action probability. Critically, the highlight in achieving the optimal rate Õ(H^2/d_mϵ^2) is our design of singleton absorbing MDP, which is a new sharp analysis tool that works with the model-based approach. We generalize such a model-based framework to the new settings: offline task-agnostic and the offline reward-free with optimal complexity Õ(H^2log(K)/d_mϵ^2) (K is the number of tasks) and Õ(H^2S/d_mϵ^2) respectively. These results provide a unified solution for simultaneously solving different offline RL problems.This work studies the statistical limits of uniform convergence for offline policy evaluation (OPE) problems with model-based methods (for episodic MDP) and provides a unified framework towards optimal learning for several well-motivated offline tasks. Uniform OPE sup_Π|Q^π-Q̂^π|<ϵ is a stronger measure than the point-wise OPE and ensures offline learning when Π contains all policies (the global class). In this paper, we establish an Ω(H^2 S/d_mϵ^2) lower bound (over model-based family) for …
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Deep learning techniques for a real-time neutrino classifier
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Ising Model Selection Using l1-Regularized Linear Regression: A Statistical Mechanics Analysis
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Computer-Aided Design as Language
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Automatic Symmetry Discovery with Lie Algebra Convolutional Network
Watch later
Nima Dehmamy, …
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%
IRM - when it works and when it doesn't: A test case of natural language inference
Watch later
Yana Dranker, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Kernelized Heterogeneous Risk Minimization
Watch later
Jiashuo Liu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%