BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Jul 24, 2023
Speakers
Silvio Savarese
Speaker · 0 followers
Steven Hoi
Speaker · 0 followers
About
The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models.This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models.BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages.The first stage bootstraps vision-language representation learning from a frozen image encoder.The second stage bootstraps vision-to-language generative learning from a frozen language model.BLIP-2 achieves state-of-the-art performance on various vision-language tasks,despite having significantly fewer trainable parameters than existing methods.For example,our model outperforms Flamingo80B by 8.7% onzero-shot VQAv2 with 54x fewer trainable parameters.We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models.This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models.BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages.The first stage bootstraps vision-language representation…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
A Picture of the Space of Typical Learnable Tasks
Watch later
Rahul Ramesh, …
Neural FIM for learning Fisher Information Metrics from point cloud data
Watch later
The Catalog Problem: Clustering and Ordering Variable-Sized Sets
Watch later
What's so special about counterfactuals?
Watch later
Health system scale language models
Watch later
Provably Invariant Learning without Domain Information
Watch later
Xiaoyu Tan, …