Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization
Jul 24, 2023
Speakers
Zejia Weng
Sprecher:in · 0 Follower:innen
Xitong Yang
Sprecher:in · 0 Follower:innen
Yu-Gang Jiang
Sprecher:in · 0 Follower:innen
About
Contrastive Language-Image Pretraining (CLIP) has demonstrated impressive zero-shot learning abilities for image understanding, yet limited effort has been made to investigate CLIP for zero-shot video recognition. We introduce Open-VCLIP, a simple yet effective approach that transforms CLIP into strong zero-shot video classifiers that can recognize unseen actions and events at test time. Our framework extends CLIP with minimal modifications to model spatial-temporal relationships in videos, making it a specialized video classifier, while striving for generalization. We formally show that training an Open-VCLIP is equivalent to continual learning with zero historical data. To address this problem, we propose Interpolated Weight Optimization, which utilizes the benefit of weight interpolation in both training and test time. We evaluate our method on three popular and challenging action recognition datasets following various zero-shot evaluation protocols and we demonstrate our approach outperforms state-of-the-art methods by clear margins. In particular, we achieve 87.9Contrastive Language-Image Pretraining (CLIP) has demonstrated impressive zero-shot learning abilities for image understanding, yet limited effort has been made to investigate CLIP for zero-shot video recognition. We introduce Open-VCLIP, a simple yet effective approach that transforms CLIP into strong zero-shot video classifiers that can recognize unseen actions and events at test time. Our framework extends CLIP with minimal modifications to model spatial-temporal relationships in videos, maki…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Paper Spotlights
Später ansehen
Andrew Ilyas, …
On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardness
Später ansehen
Haotian Ye, …
Mean-Field Analysis of Piecewise Linear Solutions for Wide ReLU Networks
Später ansehen
MG-GNN: Multigrid Graph Neural Networks for Learning Multilevel Domain Decomposition Methods
Später ansehen
Data Feedback Loops: Model-driven Amplification of Dataset Biases
Später ansehen
Rohan Taori, …
