Challenges and Opportunities in High-dimensional Variational Inference
Dec 6, 2021
Speakers
Akash Kumar Dhaka
Speaker · 0 followers
Alejandro Catalina
Speaker · 0 followers
Manushi Welandawe
Speaker · 0 followers
Jonathan Huggins
Speaker · 0 followers
Michael Andersen
Speaker · 0 followers
Aki Vehtari
Speaker · 1 follower
About
Current black-box variational inference (BBVI) methods require the user to make numerous design choices – such as the selection of variational objective and approximating family – yet there is little principled guidance on how to do so. We develop a conceptual framework and set of experimental tools to understand the effects of these choices, which we leverage to propose best practices for maximizing posterior approximation accuracy. Our approach is based on studying the pre-asymptotic tail behavior of the density ratios between the joint distribution and the variational approximation, then exploiting insights and tools from the importance sampling literature. Our framework and supporting experiments help to distinguish between the behavior of BBVI methods for approximating low-dimensional versus moderate-to-high-dimensional posteriors. In the latter case, we show that mass-covering variational objectives are difficult to optimize and do not improve accuracy, but flexible variational families can improve accuracy and the effectiveness of importance sampling – at the cost of additional optimization challenges. Therefore, for moderate-to-high-dimensional posteriors we recommend using the (mode-seeking) exclusive KL divergence since it is the easiest to optimize, and improving the variational family or using model parameter transformations to make the posterior and optimal variational approximation more similar. On the other hand, in low-dimensional settings, we show that heavy-tailed variational families and mass-covering divergences are effective and can increase the chances that the approximation can be improved by importance sampling.Current black-box variational inference (BBVI) methods require the user to make numerous design choices – such as the selection of variational objective and approximating family – yet there is little principled guidance on how to do so. We develop a conceptual framework and set of experimental tools to understand the effects of these choices, which we leverage to propose best practices for maximizing posterior approximation accuracy. Our approach is based on studying the pre-asymptotic tail beha…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
On Stochastic Stability of Deep Markov Models
Watch later
Jan Drgona, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Adversarial Reweighting for Partial Domain Adaptation
Watch later
Xiang Gu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
What Matters for Adversarial Imitation Learning?
Watch later
Manu Orsini, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Towards robust vision by multi-task learning on monkey visual cortex
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
OPT 2021: Optimization for Machine Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Pragmatic Image Compression for Human-in-the-Loop Decision-Making
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%