UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis
Dec 6, 2021
Speakers
Jianxin Ma
Speaker · 0 followers
Chang Zhou
Speaker · 0 followers
Zhikang Li
Speaker · 0 followers
Jingren Zhou
Speaker · 0 followers
Hongxia Yang
Speaker · 0 followers
About
Conditional image synthesis aims to create an image according to some multi-modal guidance in the forms of textual descriptions, reference images, and image blocks to preserve, as well as their combinations. In this paper, instead of investigating these control signals separately, we propose a new two-stage architecture, UFC-BERT, to unify any number of multi-modal controls. In UFC-BERT, both the diverse control signals and the synthesized image are uniformly represented as a sequence of discrete tokens to be processed by Transformer. Different from existing two-stage autoregressive approaches such as DALL-E and VQGAN, UFC-BERT adopts non-autoregressive generation (NAR) at the second stage to enhance the holistic consistency of the synthesized image, to support preserving specified image blocks, and to improve the synthesis speed. Further, we design a progressive algorithm that iteratively improves the non-autoregressively generated image, with the help of two estimators developed for evaluating the compliance with the controls and evaluating the fidelity of the synthesized image, respectively. Extensive experiments on a newly collected large-scale clothing dataset M2C-Fashion and a facial dataset Multi-Modal CelebA-HQ verify that UFC-BERT can synthesize high-fidelity images that comply with flexible multi-modal controls.Conditional image synthesis aims to create an image according to some multi-modal guidance in the forms of textual descriptions, reference images, and image blocks to preserve, as well as their combinations. In this paper, instead of investigating these control signals separately, we propose a new two-stage architecture, UFC-BERT, to unify any number of multi-modal controls. In UFC-BERT, both the diverse control signals and the synthesized image are uniformly represented as a sequence of discret…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Argmax Centroids With Applications to Meta and Transer Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Smooth Transfer Learning for Source-to-Target Generalization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On Locality of Local Explanation Models
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
An Empirical Study of Graph Contrastive Learning
Watch later
Yanqiao Zhu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The Benefits of Implicit Regularization from SGD in Least Squares Problems
Watch later
Difan Zou, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Profiling Pareto Front With Multi-Objective Stein Variational Gradient Descent
Watch later
Xingchao Liu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%