CLIPood: Generalizing CLIP to Out-of-Distributions
24. července 2023
Řečníci
Xingzhuo Guo
Řečník · 0 sledujících
Jialong Wu
Řečník · 0 sledujících
Ximei Wang
Řečník · 0 sledujících
Jianmin Wang
Řečník · 0 sledujících
Mingsheng Long
Řečník · 2 sledující
O prezentaci
Out-of-distribution (OOD) generalization, where the model needs to handle distribution shifts from training, is a major challenge of machine learning. Recently, contrastive language-image pre-training (CLIP) models have shown impressive zero-shot ability, revealing a promising path toward OOD generalization. However, to boost upon zero-shot performance, further adaptation of CLIP on downstream tasks is indispensable but undesirably degrades OOD generalization ability. In this paper, we aim at generalizing CLIP to out-of-distribution test data on downstream tasks. Beyond the two canonical OOD situations, domain shift and open class, we tackle a more general but difficult in-the-wild setting where both OOD situations may occur on the unseen test data. We propose CLIPood, a simple fine-tuning method that can adapt CLIP models to all OOD situations. To exploit semantic relations between classes from the text modality, CLIPood introduces a new training objective, margin metric softmax (MMS), with class adaptive margins for fine-tuning. Moreover, to incorporate both the pre-trained zero-shot model and the fine-tuned task-adaptive model, CLIPood proposes a new Beta moving average (BMA) to maintain a temporal ensemble according to Beta distribution. Experiments on diverse datasets with different OOD scenarios show that CLIPood consistently outperforms existing generalization techniques.Out-of-distribution (OOD) generalization, where the model needs to handle distribution shifts from training, is a major challenge of machine learning. Recently, contrastive language-image pre-training (CLIP) models have shown impressive zero-shot ability, revealing a promising path toward OOD generalization. However, to boost upon zero-shot performance, further adaptation of CLIP on downstream tasks is indispensable but undesirably degrades OOD generalization ability. In this paper, we aim at ge…
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization
Zhlédnout později
Zejia Weng, …
Neural FIM for learning Fisher Information Metrics from point cloud data
Zhlédnout později
The Fast Johnson-Lindenstrauss Transform Is Even Faster
Zhlédnout později
Underspecification, inductive bias, and hybrid models
Zhlédnout později
Adaptive Annealed Importance Sampling with Constant Rate Progress
Zhlédnout později
