Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPs
Dec 6, 2021
Speakers
About
Policy gradient methods can solve complex tasks but often fail when the dimensionality of the action-space or objective multiplicity grow very large. This occurs, in part, because the variance on score-based gradient estimators scales quadratically. In this paper, we address this problem through a causal baseline which exploits independence structure encoded in a novel action-target influence network. Causal policy gradients (CPGs), which follow, provide a common framework for analysing key state-of-the-art algorithms, are shown to generalise traditional policy gradients, and yield a principled way of incorporating prior knowledge of a problem domain's generative processes. We provide an analysis of the proposed estimator and identify the conditions under which variance is reduced. The algorithmic aspects of CPGs are discussed, including optimal policy factorisation, as characterised by minimum biclique coverings, and the implications for the bias-variance trade-off of incorrectly specifying the network. Finally, we demonstrate the performance advantages of our algorithm on large-scale bandit and traffic intersection problems, providing a novel contribution to the latter in the form of a spatio-causal approximation.Policy gradient methods can solve complex tasks but often fail when the dimensionality of the action-space or objective multiplicity grow very large. This occurs, in part, because the variance on score-based gradient estimators scales quadratically. In this paper, we address this problem through a causal baseline which exploits independence structure encoded in a novel action-target influence network. Causal policy gradients (CPGs), which follow, provide a common framework for analysing key stat…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Recurrent Off-policy Baselines for Memory-based Continuous Control
Watch later
Zhihan Yang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Characterizing Transformer-based Models on MPC
Watch later
Yongqin Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Unified DRO View of Multi-class Loss Functions with top-N Consistency
Watch later
Dixian Zhu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Closer Look at Gradient Estimators with Reinforcement Learning as Inference
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Second-Order Neural ODE Optimizer
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%