Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator
Nov 28, 2022
Speakers
Yinghao Xu
Speaker · 0 followers
Yujun Shen
Speaker · 0 followers
Qifeng Chen
Speaker · 0 followers
Dit-Yan Yeung
Speaker · 0 followers
About
3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. A popular solution is to adopt the generative adversarial network (GAN) and replace the generator with a 3D renderer, where volume rendering with neural radiance field (NeRF) is commonly used. Despite the advancement of synthesis quality, existing methods fail to obtain moderate 3D shapes. We argue that, considering the two-player game in the formulation of GANs, only making the generator 3D-aware is not enough. In other words, displacing the generative mechanism only offers the capability, but not the guarantee, of producing 3D-aware images, because the supervision of the generator primarily comes from the discriminator. To address this issue, we propose GeoD through learning a geometry-aware discriminator to improve 3D-aware GANs. Concretely, besides differentiating real and fake samples from the 2D image space, the discriminator is additionally asked to derive the geometry information from the inputs, which is then applied as the guidance of the generator. Such a simple yet effective design facilitates learning substantially more accurate 3D shapes. Extensive experiments on various generator architectures and training datasets verify the superiority of GeoD over state-of-the-art alternatives. Moreover, our approach is registered as a general framework such that a more capable discriminator (i.e., with a third task of novel view synthesis beyond domain classification and geometry extraction) can further assist the generator with a better multi-view consistency. Code will be made publicly available.3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. A popular solution is to adopt the generative adversarial network (GAN) and replace the generator with a 3D renderer, where volume rendering with neural radiance field (NeRF) is commonly used. Despite the advancement of synthesis quality, existing methods fail to obtain moderate 3D shapes. We argue that, considering the two-player game in the formula…
Organizer
NeurIPS 2022
Account · 953 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Meta-Learning Dynamics Forecasting Using Task Inference
Watch later
Rui Wang, …
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%
CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Certifying Some Distributional Fairness with Subpopulation Decomposition
Watch later
Mintong Kang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Artwork Spotlight: Machine Reflections: A Self-Portrait Series
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On the Double Descent of Random Features Models Trained with SGD
Watch later
Fanghui Liu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%