Neural Dubber: Dubbing for Videos According to Scripts
Dec 6, 2021
Speakers
About
Dubbing is a post-production process of re-recording actors' dialogues, which is extensively used in filmmaking and video production. It is usually performed manually by professional voice actors who read lines with proper prosody, and in synchronization with the pre-recorded videos. In this work, we propose Neural Dubber, the first neural network model to solve a novel silent video dubbing (SVD) task: synthesizing human speech synchronized with the given silent video from the text. Neural Dubber is a multi-modal text-to-speech (TTS) model that utilizes the lip movement in the video to control the prosody of the generated speech. Furthermore, an image-based speaker embedding (ISE) module is developed for the multi-speaker setting, which enables Neural Dubber to generate speech with a reasonable timbre according to the speaker's face. Experiments on the chemistry lecture single-speaker dataset and LRS2 multi-speaker dataset show that Neural Dubber can generate speech audios on par with state-of-the-art TTS models in terms of speech quality. Most importantly, both qualitative and quantitative evaluations show that Neural Dubber can control the prosody of synthesized speech by the video, and generate high-fidelity speech temporally synchronized with the video.Dubbing is a post-production process of re-recording actors' dialogues, which is extensively used in filmmaking and video production. It is usually performed manually by professional voice actors who read lines with proper prosody, and in synchronization with the pre-recorded videos. In this work, we propose Neural Dubber, the first neural network model to solve a novel silent video dubbing (SVD) task: synthesizing human speech synchronized with the given silent video from the text. Neural Dubbe…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
What Would the Expert do(.)?: Causal Imitation Learning
Watch later
Gokul Swamy, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Machine Learning and the Physical Sciences
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Data Augmentation Can Improve Robustness
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Efficient Variational Graph Autoencoders for Unsupervised Cross-domain Prerequisite Chains
Watch later
Irene Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Nearly Minimax Optimal Reinforcement Learning for Discounted MDPs
Watch later
Jiafan He, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Data Denoising and Transfer Learning in Single Cell Transcriptomics
Watch later
Nancy Zhang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%