A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning
Dec 6, 2021
Speakers
Gugan Thoppe
Speaker · 0 followers
Bhumesh Kumar
Speaker · 0 followers
About
In Multi-Agent Reinforcement Learning (MARL), multiple agents interact with a common environment and with each other, for solving a shared problem in sequential decision-making. Algorithms for MARL have a wealth of application in popular domains including gaming, robotics, and finance. In this work, we study a family of distributed nonlinear stochastic approximation schemes useful in MARL and derive a novel law of iterated logarithm. In particular, our result describes the convergence rate on almost every sample path where the algorithm converges. This result is the first of its kind in the distributed setup and provides deeper insights than the existing ones, which only discuss convergence rates in the expected or the CLT sense. Importantly, our result holds under significantly weaker assumptions: neither the gossip matrix needs to be doubly stochastic nor the stepsizes square summable. As an application, we show that, for the stepsize n^-γ with γ∈ (0, 1), the distributed TD(0) algorithm with linear function approximation has a convergence rate of 𝒪(√(n^-γlog n )) a.s.; for the 1/n type stepsize, it is 𝒪(√(n^-1loglog n)) a.s. These growth rates do not depend on the graph depicting the interactions among the different agents.In Multi-Agent Reinforcement Learning (MARL), multiple agents interact with a common environment and with each other, for solving a shared problem in sequential decision-making. Algorithms for MARL have a wealth of application in popular domains including gaming, robotics, and finance. In this work, we study a family of distributed nonlinear stochastic approximation schemes useful in MARL and derive a novel law of iterated logarithm. In particular, our result describes the convergence rate on al…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Vector-valued Gaussian Processes on Riemannian Manifolds via Gauge-Equivariant Projected Kernels
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Bayesian Adaptation for Covariate Shift
Watch later
Aurick Zhou, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Methods for Extremes
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Proximal Biasing for Bayesian Optimization and Characterization of Physical Systems
Watch later
Ryan Roussel, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Neighborhood Mixup Experience Replay: Local Convex Interpolation for Improved Sample Efficiency in Continuous Control Tasks
Watch later
Ryan Sander, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Machine Learning at Apple
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%