Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator
Nov 28, 2022
Speakers
Yinghao Xu
Speaker · 0 followers
Yujun Shen
Speaker · 0 followers
Qifeng Chen
Speaker · 0 followers
Dit-Yan Yeung
Speaker · 0 followers
About
3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. A popular solution is to adopt the generative adversarial network (GAN) and replace the generator with a 3D renderer, where volume rendering with neural radiance field (NeRF) is commonly used. Despite the advancement of synthesis quality, existing methods fail to obtain moderate 3D shapes. We argue that, considering the two-player game in the formulation of GANs, only making the generator 3D-aware is not enough. In other words, displacing the generative mechanism only offers the capability, but not the guarantee, of producing 3D-aware images, because the supervision of the generator primarily comes from the discriminator. To address this issue, we propose GeoD through learning a geometry-aware discriminator to improve 3D-aware GANs. Concretely, besides differentiating real and fake samples from the 2D image space, the discriminator is additionally asked to derive the geometry information from the inputs, which is then applied as the guidance of the generator. Such a simple yet effective design facilitates learning substantially more accurate 3D shapes. Extensive experiments on various generator architectures and training datasets verify the superiority of GeoD over state-of-the-art alternatives. Moreover, our approach is registered as a general framework such that a more capable discriminator (i.e., with a third task of novel view synthesis beyond domain classification and geometry extraction) can further assist the generator with a better multi-view consistency. Code will be made publicly available.3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. A popular solution is to adopt the generative adversarial network (GAN) and replace the generator with a 3D renderer, where volume rendering with neural radiance field (NeRF) is commonly used. Despite the advancement of synthesis quality, existing methods fail to obtain moderate 3D shapes. We argue that, considering the two-player game in the formula…
Organizer
NeurIPS 2022
Account · 953 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Cross-modal Learning for Image-Guided Point Cloud Shape Completion
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Characterizing Data Points via Second-Split Forgetting
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Get More at Once: Alternating Sparse Training with Gradient Correction
Watch later
Li Yang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Differentially Private Online-to-Batch for Smooth Losses
Watch later
Qinzi Zhang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Higher-order mutual information reveals synergistic sub-networks for multi-neuron importance
Watch later
Kenzo Clauw, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Variational Edge Partition Model for Supervised Graph Representation Learning
Watch later
Yilin He, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%