Joint Inference and input Optimization with Deep Equilibrium Models
Dec 6, 2021
Speakers
Swaminathan Gurumurthy
Speaker · 0 followers
Shaojie Bai
Speaker · 0 followers
Zico Kolter
Speaker · 2 followers
Zac Manchester
Speaker · 0 followers
About
Many tasks in deep learning involve optimizing over the inputs to a network to minimize or maximize some objective; examples include optimization over latent spaces in a generative model to match a target image, or adversarially perturbing an input to worsen classifier performance. Performing such optimization, however, is traditionally quite costly, as it involves a complete forward and backward pass through the network for each gradient step. In a separate line of work, a recent thread of research has developed the deep equilibrium (DEQ) model, a class of models that foregoes traditional network depth and instead computes the output of a network by finding the fixed point of a single nonlinear layer. In this paper, we show that there is a natural synergy between these two settings. Although, naively using DEQs for these optimization problems is expensive (owing to the time needed to compute a fixed point), we can leverage the fact that gradient-based optimization can itself be cast as a fixed point iteration to substantially improve the overall speed. That is, we show that one can simultaneously both solve for the DEQ fixed point and optimize over network inputs, all within a single ”augmented” DEQ model that jointly encodes both the original network and the optimization process. Indeed, the procedure is fast enough that it allows us to efficiently train DEQ models for tasks traditionally relying on an ”inner” optimization loop. We demonstrate this strategy on various tasks such as training generative models while optimizing over latent codes, training models for inverse problems like denoising and inpainting, and adversarial training.Many tasks in deep learning involve optimizing over the inputs to a network to minimize or maximize some objective; examples include optimization over latent spaces in a generative model to match a target image, or adversarially perturbing an input to worsen classifier performance. Performing such optimization, however, is traditionally quite costly, as it involves a complete forward and backward pass through the network for each gradient step. In a separate line of work, a recent thread of rese…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Nested Variational Inference
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On the Universality of Graph Neural Networks on Large Random Graphs
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Challenges and Opportunities Modern Approaches in Lower Resource in ML
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Kernel Functional Optimisation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
An Efficient Transfer Learning Framework for Multiagent Reinforcement Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Dynamic Causal Bayesian Optimization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%