All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference

Jul 12, 2020



While the Evidence Lower Bound (ELBO) has become a ubiquitous objective for variational inference, the recently proposed Thermodynamic Variational Objective (TVO) leverages thermodynamic integration to provide a tighter and more general family of bounds. In previous work, the tightness of these bounds was not known, grid search was used to choose a `schedule' of intermediate distributions, and model learning suffered with ostensibly tighter bounds. We interpret the geometric mixture curve common to TVO and related path sampling methods using the geometry of exponential families, which allows us to characterize the gap in TVO bounds as a sum of KL divergences along a given path. Further, we propose a principled technique for choosing intermediate distributions using equal spacing in the moment parameters of our exponential family. We demonstrate that this scheduling approach adapts to the shape of the integrand defining the TVO objective and improves overall performance. Additionally, we derive a reparameterized gradient estimator which empirically allows the TVO to benefit from additional, well chosen partitions. Finally, we provide a unified framework for understanding thermodynamic integration and the TVO in terms of Taylor series remainders.



About ICML 2020

The International Conference on Machine Learning (ICML) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence known as machine learning. ICML is globally renowned for presenting and publishing cutting-edge research on all aspects of machine learning used in closely related areas like artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, and robotics. ICML is one of the fastest growing artificial intelligence conferences in the world. Participants at ICML span a wide range of backgrounds, from academic and industrial researchers, to entrepreneurs and engineers, to graduate students and postdocs.

Store presentation

Should this presentation be stored for 1000 years?

How do we store presentations

Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%


Recommended Videos

Presentations on similar topic, category or speaker