Controllable and Compositional Generation with Latent-Space Energy-Based Models
Dec 6, 2021
Speakers
Arash Vahdat
Speaker · 0 followers
Anima Anandkumar
Speaker · 1 follower
About
Controllable generation is one of the key requirements for successful adoption of deep generative models in real-world applications, but it still remains challenging. In particular, the compositional ability to generate novel concept combinations is out of reach for most current models. In this work, we use energy-based models (EBMs) to handle compositional generation over a set of attributes. To make them scalable to high-resolution image generation, we introduce an EBM in the latent space of a pre-trained generative model such as StyleGANs. We propose a novel EBM formulation representing the joint distribution of data and attributes together, and we show how sampling from it is formulated as solving an ordinary differential equation (ODE). Given a pre-trained generator, all we need for controllable generation is to train an attribute classifier. Sampling with ODEs is done efficiently in the latent space and is robust to hyperparameters. Thus, our method is simple, fast to train, and efficient to sample. Experimental results show that our method outperforms the state-of-the-art in both conditional sampling and sequential editing. In compositional generation, our method excels at zero-shot generation of unseen attribute combinations. Also, by composing energy functions with logical operators, this work is the first to achieve strong compositionality in generating photo-realistic images of resolution 1024x1024.Controllable generation is one of the key requirements for successful adoption of deep generative models in real-world applications, but it still remains challenging. In particular, the compositional ability to generate novel concept combinations is out of reach for most current models. In this work, we use energy-based models (EBMs) to handle compositional generation over a set of attributes. To make them scalable to high-resolution image generation, we introduce an EBM in the latent space of a…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Indigenous in AI Workshop
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
SWR using quantum - ML
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Statistical Query Lower Bounds for List-Decodable Linear Regression
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Introduction to the Workshop on Metacognition in the Age of AI: Challenges and Opportunities
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Multi-dimensional data Completion via Low-Rank Tensor Respresentation under Coupled Transform
Watch later
Jian-Li Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Generalisation in Lifelong RL via Logical Composition
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%