CLIPood: Generalizing CLIP to Out-of-Distributions
24. července 2023
Řečníci
Xingzhuo Guo
Sprecher:in · 0 Follower:innen
Jialong Wu
Sprecher:in · 0 Follower:innen
Ximei Wang
Sprecher:in · 0 Follower:innen
Jianmin Wang
Sprecher:in · 0 Follower:innen
Mingsheng Long
Sprecher:in · 2 Follower:innen
O prezentaci
Out-of-distribution (OOD) generalization, where the model needs to handle distribution shifts from training, is a major challenge of machine learning. Recently, contrastive language-image pre-training (CLIP) models have shown impressive zero-shot ability, revealing a promising path toward OOD generalization. However, to boost upon zero-shot performance, further adaptation of CLIP on downstream tasks is indispensable but undesirably degrades OOD generalization ability. In this paper, we aim at generalizing CLIP to out-of-distribution test data on downstream tasks. Beyond the two canonical OOD situations, domain shift and open class, we tackle a more general but difficult in-the-wild setting where both OOD situations may occur on the unseen test data. We propose CLIPood, a simple fine-tuning method that can adapt CLIP models to all OOD situations. To exploit semantic relations between classes from the text modality, CLIPood introduces a new training objective, margin metric softmax (MMS), with class adaptive margins for fine-tuning. Moreover, to incorporate both the pre-trained zero-shot model and the fine-tuned task-adaptive model, CLIPood proposes a new Beta moving average (BMA) to maintain a temporal ensemble according to Beta distribution. Experiments on diverse datasets with different OOD scenarios show that CLIPood consistently outperforms existing generalization techniques.Out-of-distribution (OOD) generalization, where the model needs to handle distribution shifts from training, is a major challenge of machine learning. Recently, contrastive language-image pre-training (CLIP) models have shown impressive zero-shot ability, revealing a promising path toward OOD generalization. However, to boost upon zero-shot performance, further adaptation of CLIP on downstream tasks is indispensable but undesirably degrades OOD generalization ability. In this paper, we aim at ge…
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
InfoDiffusion: Representation Learning Using Information Maximizing Diffusion Models
Später ansehen
Efficient displacement convex optimization with particle gradient descent
Später ansehen
CircuitNet: A Generic Neural Network to Realize Universal Circuit Motif Modeling
Später ansehen
Yansen Wang, …
Policy Gradient in Robust MDPs with Global Convergence Guarantee
Später ansehen
Qiuhao Wang, …
Whose Opinions Do Language Models Reflect?
Später ansehen
