Multi-Lingual Acquisition on Multimodal Pre-training for Cross-modal Retrieval
Nov 28, 2022
Speakers
About
Vision and diverse languages are important information sources in our living world. A model that understands multi-modalities and multi-languages can be applied to a wider range of real-life scenarios. To build such a multimodal and multilingual model, existing works try to ensemble vision-language data from multiple languages in pre-training. However, due to the large number of languages, these works often require huge computing resources and cannot be flexibly extended to new languages. In this work, we propose a MultiLingual Acquisition (MLA) framework that can easily empower a monolingual Vision-Language Pre-training (VLP) model with multilingual capability. Specifically, we design a lightweight language acquisition encoder based on state-of-the-art monolingual VLP models. We further propose a two-stage training strategy to optimize the language acquisition encoder, namely the Native Language Transfer stage and the Language Exposure stage. With much less multilingual training data and computing resources, our model achieves state-of-the-art performance on multilingual image-text and video-text retrieval benchmarks.Vision and diverse languages are important information sources in our living world. A model that understands multi-modalities and multi-languages can be applied to a wider range of real-life scenarios. To build such a multimodal and multilingual model, existing works try to ensemble vision-language data from multiple languages in pre-training. However, due to the large number of languages, these works often require huge computing resources and cannot be flexibly extended to new languages. In thi…
Organizer
NeurIPS 2022
Account · 958 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Deep Learning for Rapid Landslide Detection using Synthetic Aperture Radar (SAR) Datacubes
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Ranking Game for Imitation Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
You Can’t Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Sensor network and spatiotemporal time series
Watch later
Andrea Cini, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Protein structure generation via folding diffusion
Watch later
Kevin E. Wu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Functional Ensemble Distillation
Watch later
Coby Penso, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%