Towards A Unified Policy Abstraction Theory and Representation Learning Approach in Markov Decision Processes
Dec 2, 2022
Speakers
Hongyao Tang
Speaker · 0 followers
Jianye Hao
Speaker · 0 followers
About
Lying on the heart of intelligent decision-making systems, how policy is represented and optimized is a fundamental problem. The root challenge in this problem is the large scale and the high complexity of policy space, which exacerbates the difficulty of policy learning especially in real-world scenarios. Towards a desirable surrogate policy space, recently policy representation in a low-dimensional latent space has shown its potential in improving both the evaluation and optimization of policy. The key question involved in these studies is by what criterion we should abstract the policy space for desired compression and generalization. However, both the theory on policy abstraction and the methodology on policy representation learning are less studied in the literature. In this work, we make very first efforts to fill up the vacancy. First, we propose a unified policy abstraction theory, containing three types of policy abstraction associated to policy features at different levels. Then, we generalize them to three policy metrics that quantify the distance (i.e., similarity) of policies, for more convenient use in learning policy representation. Further, we propose a policy representation learning approach based on deep metric learning. For the empirical study, we investigate the efficacy of the proposed policy metrics and representations, in characterizing policy difference and conveying policy generalization respectively. Our experiments are conducted in both policy optimization and evaluation problems, containing trust-region policy optimization (TRPO), diversity-guided evolution strategy (DGES) and off-policy evaluation (OPE). Somewhat naturally, the experimental results indicate that there is no a universally optimal abstraction for all downstream learning problems; while the influence-irrelevance policy abstraction can be a generally preferred choice.Lying on the heart of intelligent decision-making systems, how policy is represented and optimized is a fundamental problem. The root challenge in this problem is the large scale and the high complexity of policy space, which exacerbates the difficulty of policy learning especially in real-world scenarios. Towards a desirable surrogate policy space, recently policy representation in a low-dimensional latent space has shown its potential in improving both the evaluation and optimization of policy…
Organizer
NeurIPS 2022
Account · 961 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Riemannian Score-Based Generative Modeling
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
When Combinatorial Thompson Sampling meets Approximation Regret
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
MaskPlace: Fast Chip Placement via Reinforced Visual Representation Learning
Watch later
Yao Lai, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning Discrete Directed Acyclic Graphs via Backpropagation
Watch later
Andrew Wren, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Group Meritocratic Fairness in Linear Contextual Bandits
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%