TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation
Dec 6, 2021
Speakers
Tanzila Rahman
Speaker · 0 followers
Mengyu Yang
Speaker · 0 followers
Leonid Sigal
Speaker · 0 followers
About
The recent success of transformer models in language, such as BERT, has motivated the use of such architectures for multi-modal feature learning and tasks. However, most multi-modal variants (e.g., ViLBERT) have limited themselves to visual-linguistic data. Relatively few have explored its use in audio-visual modalities, and none, to our knowledge, illustrate them in the context of granular audio-visual detection or segmentation tasks such as sound source separation and localization. In this work, we introduce TriBERT – a transformer-based architecture, inspired by ViLBERT, which enables contextual feature learning across three modalities: vision, pose, and audio, with the use of flexible co-attention. The use of pose keypoints is inspired by recent works that illustrate that such representations can significantly boost performance in many audio-visual scenarios where often one or more persons are responsible for the sound explicitly (e.g., talking) or implicitly (e.g., sound produced as a function of human manipulating an object). From a technical perspective, as part of the TriBERT architecture, we introduce a learned visual tokenization scheme based on spatial attention and leverage weak-supervision to allow granular cross-modal interactions for visual and pose modalities. Further, we supplement learning with sound-source separation loss formulated across all three streams. We pre-train our model on the large MUSIC21 dataset and show improved performance in audio-visual sound source separation on that dataset as well as other datasets through fine-tuning. In addition, we show that the learned TriBERT representations are generic and significantly improve performance on other audio-visual tasks such as cross-modal audio-visual-pose retrieval by as much as 66.7The recent success of transformer models in language, such as BERT, has motivated the use of such architectures for multi-modal feature learning and tasks. However, most multi-modal variants (e.g., ViLBERT) have limited themselves to visual-linguistic data. Relatively few have explored its use in audio-visual modalities, and none, to our knowledge, illustrate them in the context of granular audio-visual detection or segmentation tasks such as sound source separation and localization. In this wor…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Private Non-smooth ERM and SCO in Sub-quadratic Steps
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Generalized Proximal Policy Optimization with Sample Reuse
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Proximal Causal Inference
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
Watch later
Robert Logan, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Hyperparameter Tuning is All You Need for LISTA
Watch later
Xiaohan Chen, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Neural Active Learning with Performance Guarantees
Watch later
Zhilei Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%