Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator
Nov 28, 2022
Speakers
Yinghao Xu
Speaker · 0 followers
Yujun Shen
Speaker · 0 followers
Qifeng Chen
Speaker · 0 followers
Dit-Yan Yeung
Speaker · 0 followers
About
3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. A popular solution is to adopt the generative adversarial network (GAN) and replace the generator with a 3D renderer, where volume rendering with neural radiance field (NeRF) is commonly used. Despite the advancement of synthesis quality, existing methods fail to obtain moderate 3D shapes. We argue that, considering the two-player game in the formulation of GANs, only making the generator 3D-aware is not enough. In other words, displacing the generative mechanism only offers the capability, but not the guarantee, of producing 3D-aware images, because the supervision of the generator primarily comes from the discriminator. To address this issue, we propose GeoD through learning a geometry-aware discriminator to improve 3D-aware GANs. Concretely, besides differentiating real and fake samples from the 2D image space, the discriminator is additionally asked to derive the geometry information from the inputs, which is then applied as the guidance of the generator. Such a simple yet effective design facilitates learning substantially more accurate 3D shapes. Extensive experiments on various generator architectures and training datasets verify the superiority of GeoD over state-of-the-art alternatives. Moreover, our approach is registered as a general framework such that a more capable discriminator (i.e., with a third task of novel view synthesis beyond domain classification and geometry extraction) can further assist the generator with a better multi-view consistency. Code will be made publicly available.3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. A popular solution is to adopt the generative adversarial network (GAN) and replace the generator with a 3D renderer, where volume rendering with neural radiance field (NeRF) is commonly used. Despite the advancement of synthesis quality, existing methods fail to obtain moderate 3D shapes. We argue that, considering the two-player game in the formula…
Organizer
NeurIPS 2022
Account · 953 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Your Out-of-Distribution Detection Method is Not Robust!
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On Translation and Reconstruction Guarantees of the Cycle-Consistent Generative Adversarial Networks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
FLAIR: Federated Learning Annotated Image Repository
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning Multi-resolution Functional Maps with Spectral Attention for Robust Shape Matching
Watch later
Lei Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Informative Rewards and Generalization in Curriculum Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%