Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator
Dec 6, 2022
Speakers
Yinghao Xu
Speaker · 0 followers
Yujun Shen
Speaker · 0 followers
Qifeng Chen
Speaker · 0 followers
Dit-Yan Yeung
Speaker · 0 followers
About
3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. A popular solution is to adopt the generative adversarial network (GAN) and replace the generator with a 3D renderer, where volume rendering with neural radiance field (NeRF) is commonly used. Despite the advancement of synthesis quality, existing methods fail to obtain moderate 3D shapes. We argue that, considering the two-player game in the formulation of GANs, only making the generator 3D-aware is not enough. In other words, displacing the generative mechanism only offers the capability, but not the guarantee, of producing 3D-aware images, because the supervision of the generator primarily comes from the discriminator. To address this issue, we propose GeoD through learning a geometry-aware discriminator to improve 3D-aware GANs. Concretely, besides differentiating real and fake samples from the 2D image space, the discriminator is additionally asked to derive the geometry information from the inputs, which is then applied as the guidance of the generator. Such a simple yet effective design facilitates learning substantially more accurate 3D shapes. Extensive experiments on various generator architectures and training datasets verify the superiority of GeoD over state-of-the-art alternatives. Moreover, our approach is registered as a general framework such that a more capable discriminator (i.e., with a third task of novel view synthesis beyond domain classification and geometry extraction) can further assist the generator with a better multi-view consistency. Code will be made publicly available.3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. A popular solution is to adopt the generative adversarial network (GAN) and replace the generator with a 3D renderer, where volume rendering with neural radiance field (NeRF) is commonly used. Despite the advancement of synthesis quality, existing methods fail to obtain moderate 3D shapes. We argue that, considering the two-player game in the formula…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Generalized Delayed Feedback Model with Post-Click Information in Recommender Systems
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Model Zoo: A Dataset of Diverse Populations of Neural Network Models
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Dungeons and Data: A Large Scale NetHack Dataset
Watch later
Eric Hambro, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Graph Neural Networks are Dynamic Programmers
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Flexible Neural Image Compression via Code Editing
Watch later
Chenjian Gao, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Keynote 4: Designing AI Systems for Digital Well-Being
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%